Hyperbolicity

Simple smoothing of demand with shelf life

2018-10-10T04:00:01+00:00

A very simple discrete algorithm for smoothing out impulses to a target constant value

Demand smoothing

Let’s assume you use a lot of toilet paper at your home. Some number of rolls per week(integer rolls). If you buy new packages of toilet paper once a month, let’s figure out a simple algorithm to keep your bathroom well-stocked.

Roll up weekly data

Each week, we have our usage, but since we’re only buying once every month we need to consider the sum over the weeks inside. Consider M-S weeks, then the months have the following number of Sundays:

[4,4,4,5,4,4,5,4,5,4,4,5]

We’ll return to this later.

Available rolls

We dont want to run out, period. As not-a-risk taker, and someone who occasionally throws parties, I want at least twice as much TP around as I normally need. Let’s then say whatever my biggest usage per week is, times two, will be my target t.

Minimizing Error

Because we have a target for available rolls, our goal throughout the month, is to have our order keep us as close to that value as possible.

For i, \(1 ≤ i ≤ 5\) the numbered weeks in the month, and u_i the corresponding TP usages. Also assume that before you buy some more TP, you have o the “on shelf” TP from the month before. We’ll be calculating our TP purchase x

We wish to minimize

\[\mathop{\arg\,\min}\limits_x\left(\sum_i\left(\mid x+o-\sum^i_j u_j -t \mid\right)\right)\]

This feels like something that should be somewhat easy to minimize, and to a certain extent it is. It relies on a little lemma though:

Little Lemma

For a set of numbers S, and a value Y

\[\mathop{\arg\,\min}\limits_{s \in S}\left(\sum_i\left(\mid S - Y \mid\right)\right) = median\left(S\right)\]

So let \(Y= x+o-t\), and consider instead, the partial sums \(U_i = \sum^i_j u_j\), then

\[\mathop{\arg\,\min}\limits_x\left(\sum_i\left(\mid x+o-\sum^i_j u_j -t \mid\right)\right) = \mathop{\arg\,\min}\limits_{U_i}\left(\sum_i\left(\mid U_i - Y \mid\right)\right) = median\left(U_i\right)\]

Ap-PLY-ing our algorithm

To compute a year’s worth of TP purchases, we need now apply the previous. Let’s start by assuming our TP consumption is exactly last years(broken into months):

weekly_usage = [
  [4,3,2,4,],
  [3,3,2,4,],
  [3,5,3,2,],
  [4,5,3,3,2,],
  [4,3,3,2,],
  [6,4,3,3,],
  [2,7,4,3,3,],
  [4,2,4,3,],
  [3,6,2,4,3,],
  [3,2,11,2,],
  [4,3,7,3,],
  [2,4,3,3,5,],
]

Now we need to send each of these little lists into the median calculator:

partial_sum = lambda a, b: a + [a[-1] + b]

def take_median_or_upper_bound_of_partial_sums(list_of_ints):
  return reduce(
    partial_sum,
    list_of_ints[1:],
    list_of_ints[0:1],
  )[len(list_of_ints)/2 if len(list_of_ints) > 2 else -1]

And then we can just run through our weekly usage in blocks of months:

monthly_purchases, start_of_month_availability = [], incoming_availability
for month in weekly_usage:
  necessary = max(
    take_median_or_upper_bound_of_partial_sums(month) -
    start_of_month_availability +
    (target or 0),
    0
  )
  monthly_purchases.append(necessary)
  start_of_month_availability = necessary + start_of_month_availability - sum(month)

A gotcha

If you dont start exactly when you’re buying, every week that passes before you buy needs to be subtracted from the start_of_month_availability to start with:

start_of_month_availability = incoming_availability - used_before_first_order

Example

As an example, with the data above, you can compute the purchase amounts. In this case let’s assume the incoming_availability is 12 and our target is twice the biggest week’s usage: 22, it looks like this:

Available TP vs. our Target

purchase: 19
current available: 31
current available: 27
current available: 24
current available: 22
purchase: 12
current available: 30
current available: 27
current available: 24
current available: 22
purchase: 15
current available: 33
current available: 30
current available: 25
current available: 22
purchase: 14
current available: 34
current available: 30
current available: 25
current available: 22
current available: 19
purchase: 15
current available: 32
current available: 28
current available: 25
current available: 22
purchase: 15
current available: 35
current available: 29
current available: 25
current available: 22
purchase: 16
current available: 35
current available: 33
current available: 26
current available: 22
current available: 19
purchase: 16
current available: 32
current available: 28
current available: 26
current available: 22
purchase: 14
current available: 33
current available: 30
current available: 24
current available: 22
current available: 18
purchase: 23
current available: 38
current available: 35
current available: 33
current available: 22
purchase: 16
current available: 36
current available: 32
current available: 29
current available: 22
purchase: 12
current available: 31
current available: 29
current available: 25
current available: 22
current available: 19

Simple smoothing of demand with shelf life was originally published by Bryan Bischof at Hyperbolicity on October 10, 2018.

The route to the top 100

2018-01-01T04:00:01+00:00

The shortest¹ cycling route that climbs the top 100 climbs in the continental US.

The idea

A while back, Randal Olson posted a blog post about the optimal road trip across the US. The parameters of his project were to design a car trip that touched all the states, and he used landmarks in each state to make it more interesting.

Two things really stood out about this to me:

he open-sourced his notebook that wrapped up some of more annoying API/wrapper stuff and the genetic algorithm
he noted that the API accepted a bike routes flag!

I was inspired! In addition to a mathy, I’m a serious lover of cycling. My project, was essentially:

Combine the dataset of Top 100 road cycling climbs with Randal Olson’s project to achieve the minimal bike route that includes all these climbs.

The Top 100 road climbs is a nice project by pjammcycling that rank the hardest climbs in the US(as defined by Fiets index).

Note: I wanted this to be a cycling route, no usage of boats allowed. So Hawaii was out. Unfortunately—and somewhat surprisingly, Hawaii has 7 of the top 100 road climbs in the US. Luckily, the list provided 107 climbs, so…

A little about the math

More details are in Randy’s post, but I’ll provide a super short explanation of the idea.

This is the famous Traveling Salesman Problem which finds the order that minimizes the distance to visit a collection of sites.

In our case, the starts of the climbs are the sites. We use Google Maps API to find cycling routes between each pair of sites(that’s 10000 measurements).

Once you have all the pairwise distance it would require n! or 10000!(this is over 35000 digits) different orderings you’d have to try. We use something called a genetic algorithm which picks a random order, changes some and checks if it improves or worsens. After many many iterations it comes to a good, but not necessarily perfect solution¹.

Some challenges

Initially, I was going to grab the list of locations names from pjamm, run them through his notebook, and let it crunch. But I was wary that things weren’t going to be easy, because the names of the locations weren’t very accurate.

My first round yielded a route that the Google Maps visualization API couldn’t plot because many of the locations it couldn’t find anything for. I switched to (lat, long) after a few small modifications to his notebook. This worked much better at getting me close to the initial points for each of the climbs.

Again, the visualization code died on this route. This time, it was a mixture of some bad waypoints, and a few other things:

the route was longer than the maximum alloted
some locations were on roads that are not bike friendly so Google maps disliked this
some locations were connected by seasonal roads, so Google maps disliked this (this issue returns later)

Google Maps visualizations

I managed to do some small adjustments and finally get an image for this stage of the visualization, knowing that there was much more to do. I also broke the route up into ten pieces, anticipating the difficulties due to scale.

The long first route.

Plots of route starts via Google Maps.

The last ten.

Transitioning to an actual cycling route

I always wanted this to turn into a cycling route in the end, and the numerous issues with the Google maps visualization made clear that things wouldn’t go smoothly. About this time I came to some scary realizations:

the data that I had didn’t include the actual climbs, they were simply latitude and longitude of the starting location
I didn’t have the Strava segments associated to each of these climbs
I couldn’t find a route builder that accepted lists of coordinates to build a route, and Strava doesn’t allow adding waypoints later, or searching by lat/long in their route builder.

At this point, I took a major step back. The relatively naive approach I was taking had come to an end. I then started working on building more of a decent dataset.

The dataset

Until this moment I’d been able to simply copy the table off the front page (Note that the website has changed a bit recently) , and with some text-editor dexterity, turn it into something that looked like a data set. Now, I knew I needed to do a few things:

scrape the dataset and make sure it was clean and more easily referenced
download the Strava segments for each climb and add the segment number to my dataset

These were relatively straightforward, here’s how I grabbed the segments:

printf "1, " >> enum_outputstrings.txt
curl -s "https://www.pjammcycling.com/1.--mauna-kea---test2.html" | grep www.Strava | awk 'BEGIN {FS="src="} {for(i=2;i<=NF;i++)print $i}' | cut -d ">" -f 1 | tr -d \'\" | sed -e 's/\/embed$//'  >> enum_outputstrings.txt

I was feeling good about this and started the next step—which I’ll explain in a minute—before I realized a huge problem with this.

If you look closely at how I’m extracting the above, I’m using a few tricks about the order that I download them, to infer the number in the ranking, and then searching that page for where I see the string www.Strava. Why is this a problem? Because pjammcycling has changed the order without updating the urls… Also, my original code wasn’t clever enough to predict pages with multiple Strava links.

Both of these meant that this data extract was quite seriously erroneous. This wouldn’t have been particularly awful, had I been using a more sophisticated method for this data extraction. The subtlety really ended up being the killer. I ended up not really updating the shell script, and instead just manually adjusted things, or used some text-editor shenanigans.

This was the first part of the problem that pushed the timeline way back.

Plotting the route

I still hadn’t found a solution to either of my plotting problems. So, I thought to myself:

“I’ll buckle down and input all these lat/long by hand”

Luckily, I had the foresight to split them into ten-climb groups. It took about an hour, but I build the original ten routes, by copy-pasting lat/long pairs into ridewithGPS.

A note about ridewithGPS: I think that I’d more naturally use Strava to build the route, because all of the climbs have the effort data, but ridewithGPS’ route tool is hands-down easier to use, and allows for addition of waypoints, splitting, combining, etc. I’m not confident this project would be possible with Strava’s route tool, but doable with ridewithGPS’.

So now I have ten long routes that connect the starting points of each climb, but I’m a long way from the actual goal. This came after some manual adjustment here and there, where the points were problematic as I previously mentioned.

A brief aside about loops

The final product is not a loop. This is completely intended. The output of the algorithm is a loop. I didn’t really like that, so I decided I would break it. Because I was breaking it, I could choose which pair of points didn’t need connection. Keeping in line with the original goal of the project, I removed the longest distance connection between any two consecutive points in the optimized path.

This turned out to be the connection from northwest Wyoming to upstate New York. I arbitrarily decided on the orientation of the route, which mean that upstate New York was the start, and that it would finish miles from the Wyoming/Montana border.

Drag on

I had a moment of realization as I was finishing up the manual route building of the ten original segments where I thought

“I’m going to have to drag the path to make all the hill climbs part of this route—that’s going to suck”

It did.

I used a timer during this project and this part of the project took approximately 8 hrs. This involved:

downloading a chrome extension that can open lists of urls as tabs
using my dataset of Strava segments to visually detail what the routes look like
dragging a point just after the starting waypoint to the top of the climb
adjusting points in-between to make sure they followed the exact path as the specified segment
fixing some data issues again (yes, there were more at this point, c.f. below)
adjusting the route between ends of climbs and starts of next to optimize both: enjoyability of ride, and minimal distance taking into consideration the different location

This last point was important to me. The genetic algorithm provides a pretty good global solution, but since I was effectively doubling the number of waypoints, and the algorithm never saw the new ones, there was only this human option.

This process was very challenging, and I had to spread this over a few weeks. In some cases, the route builder seemed insistant on not following the route I wanted and occasionally would require six or more extra points to force it. Not to mention the incredible frustration of summer roads. The astute reader may realize that we’ve been talking about mountains here, which in some cases, get snowy. In places like Colorado and some in the Sierras, the climb is on a summer road that I for the life of me, could not get ridewithGPS to plot on. However, again we see the flexibility of their tool: you can freehand.

So yes, dear reader, a few of the climbs are freehanded(poorly I might add).

More data issues

How could there be any data issues left by now? …Sigh…

In one case, I had apparently deleted a digit in the dataset accidentally. In two cases, the Strava segment listed was no longer listed on Strava. And more! These little issues eat up time and make an already exhausting task feel impossible. Nevertheless, I persisted.

Explicit changes to the dataset

Here is a short list of data changes that I made:

A few climbs had double Strava segments
Three climbs had bad coords for the starts: 44, 61, 62
A few had no Strava segment listed, so I went and found some using the segment explorer
Hwy 21-245-180-198 California, doesn’t have a Strava route, but there’s a longer one that can be used
Tollhouse Rd had the wrong route number, it’s actually 9887218
The Gibraltar Rd segment has been flagged, so I used the Gibraltar TT segment(shorter), but on the top 100 route, it still does the longer version
Fixed the Lone Pine to Table Mountain segment to a better one

One for all

Now I had ten routes (1, 2, 3, 4, 5, 6, 7, 8, 9, 10), all with ten climbs each. Again, ridewithGPS to the rescue; combining was relatively painless. Although, I’ll admit that it’s a little hard on the browser to have close to 300 waypoints stretched over 14000 miles of route, unsurprisingly.

💯💯💯 So I guess, it’s time to show it to you!!!? 💯💯💯

^^^^^^ Click to explore the interactive route on ridewithGPS ^^^^^^

By the numbers

100 climbs in the Continental US
17 states
14235 miles
965210 ft of elevation

Some technical notes

The dataset was extracted originally in November, 2017. I wont likely be updating the routes.
The actual route doesn’t change as the ranking list changes, but, obviously, it will be effected if totally new routes are added.
Bear Camp was not in the dataset when the original dataset was built, hence it’s not included. Here is the route. It should be in this section of the route, before you climb Mt. Ashland the Strava route for the new Bear Camp is here
As of just yesterday, the pjammcycling website has changed dramatically. Additionally, they’ve expanded to worldwide now 😏.

Next Steps

First, I want to mention an accidental offshoot of this project: maximal sub-routes. By this I mean, “what are some sections of this route which maximize the elevation/mileage trade-off?” These are things that you could consider turning into crazy Gran-Fondos like the California Triple Crown series which are hard-as-nails double centuries. There are some obvious ones in this data-set, and some manual inspection can find some cute little things like Mile 310-510 in the 4th route which contains almost 30k climbing on six of the top 100 climbs! (HRS I’m looking at you here!)

I’ll be following this post up with some fun dives into snippets like the above.

Finally, I want to mention that this project, believe it or not, is part one. To keep the mystery alive, I’ll only mention that part two is to compute the Best of the Top 100, and more generally a StraVaELO…

The route to the top 100 was originally published by Bryan Bischof at Hyperbolicity on January 01, 2018.

Contiguous letter sequences

2017-09-10T04:00:01+00:00

What’s the shortest word that contains ‘ABCDE’ and so on…

Contiguous letter sequences

Haggard Hawks—one of my favorite Twitter accounts, and the subject of a different project once before—tweeted out this fun fact.

GOLDFINCHES contains the letters CDEFGHI.
JASMINELIKE contains the letters IJKLMN.
PROPINQUITIES contains the letters OPQRST.

Which got the ol’ thinker thinkin’.

The code for this is relatively straightforward, so I wont belabor it, but I will admit I had to think a bit about what exactly I wanted to do. I came up with

For each subsequence of the alphabet, what is the shortest word in my dictionary(we'll come back to this) that contains all letters in this subsequence?

A few points:

I wanted the shortest word as a way to make the solutions more unique, and more “purely” about the letters.
I wanted to consider all subsequences because I predicted it would make for a better visualization and decomposition of the alphabet than random splittings of letters.

The coding part

The most challenging of the coding component was an organized and reasonably efficient way to run through all words in the dictionary and all subsequences. Since the dictionary is long in comparison to the number of subsequences—(n^2 + n) / 2)-n=325 because I ignore single letters—I knew that I’d want to scroll through the dictionary and check which of the subsequences were there. Thus I made a dict, with keys all the subsequences and values as empty lists, and then checked if the alphabetized-uniqued version of the word contained the subsequence. If so, I added it to the dictionary in length order. This worked and wasn’t too slow, even for 300k words.

If I were really flexing, I’d have made the subsequences into a trie, and traversed that to reduce the number of checks, but it didn’t seem worth it since the run-time was so small already.

Visualization

After running through the above, I knew in my mind that I wanted some matrix presentation of the data, but for the life of me couldn’t make sense of how to do it. I kept thinking that the rows and columns would be letters contained in the sequence, but this actually becomes sort of a tensor(3-dimensional matrix), in that there is a matrix of containment for every pair of letters. After far too long I realized that starting letter and ending letter uniquely identify the subsequence, and realized what I wanted.

I used Tableau to simplify generating the graphics, which for this simple of a project, was plenty.

Two matrices for two differently sized dictionaries

In the above you’ll notice two matrices because I used two separate word lists. Originally, I just used the unix wordlist, but I immediately noticed I didn’t find JASMINELIKE in my analysis, or even another word that contained IJKLMN. This bothered me, so I looked around for a longer wordlist similar to the unix one. I found this one. I liked how similar the wordlists were, and this one is about 50% larger. You’ll notice that the larger dictionary does, in-fact, have jasminelike, and adds some other good words on the outskirts of this matrix.

I liked that the big stair steps in the matrices show where certain letters have a lot less words.

Efficiency

Because I was thinking about how short the words were in terms of their subsequence, I decided to compute efficiency of the words. Hueristically, this meant how many letters beyond the subsequence were necessary to create the word. We expect that as the subsequence grows in length, the efficiency will also decrease, and that was true.

Efficiency matrix for large dictionary

To compute this I used the following formula:

(
  1-([Containing Word Len]-[number of consecs])/[Containing Word Len]
)*[number of consecs]

Which is the percentage of the number of unnecissary letters multiplied by the number of consecutive letters. The reason for skewing by consecutive letters is to accomidate that it’s harder and harder to make a word as the number of letters increase.

Some analysis

If I wanted to recreate the original tweet with the shortest possible words:

LIGHTFACED is a shorter word for CDEFGHI
JASMINELIKE is the shortest for IJKLMN
PAROQUETS is shorter for OPQRSTU

If I wanted the shortest total letters to express the alphabet:

AB,CD,FE,HG,JI,KL,MN,PO,QR,ST,VU,XW,YEZ

Which isn’t very interesting, so I switched to shortest number of words and minimizing letters within that:

BOLDFACE,HIGHJACK,PLASMOQUIN,LIVERWURST,XYZ

Note that this required a bit more coding, to find the optimal sequence.

Contiguous letter sequences was originally published by Bryan Bischof at Hyperbolicity on September 10, 2017.

Puzzles

2017-05-19T04:00:01+00:00

A place for my continued toying with puzzles. 538 puzzles, WACan’t, and others.

Finnegan’s Quaternions

The ever distracting Wolfram Alpha Can't Twitter account, ensnared me a few days ago with this tweet

Which asked for all the is, js, and ks from Finnegan’s wake, to be taken in order to be a quaternionic product. The problem wasn’t the most fascinating, but just silly enough.

Here’s a gist of my solution. I definitely think that the multiplication function could be more elegant, but c’est la vie. I was happy that I thought to use reduce.

Billiards racks

Another distracting Twitter account is solve my maths, who retweeted this little puppy.

This problem was a little meatier. It asks if you can rearrange the pool balls in a billiards rack such that each ball is the difference of the two balls below it(excluding the bottom row). I liked how this problem immediately starts suggesting some data structures.

My mental model for how to construct a rack was to start in the lower-left corner, and add a ball to the bottom row at a time, and stop when it becomes impossible. Then, running through all the ball orderings, will return only the racks that make it all the way.

So I started with a row object, and then a rack object. The rows have append, and the racks have add_ball. The rack needs fill_last because it didn’t cleanly fit into the paradigm of add_ball; also added a print_rack function for obvious reasons.

The logic is straightforward, and starts with two balls:

def __init__(self, leftmost, secondleftmost, operation=delta, rack_size=5):
    if (leftmost != secondleftmost and leftmost != 2*secondleftmost and 2*leftmost != secondleftmost):
        self.operation=operation
        self.rack_size=rack_size
        self.rows = [row(rack_size, leftmost)]
        self.rows[0].append(secondleftmost)
        self.remaining_balls = list(range(1,((rack_size*(rack_size+1))/2)+1))
        self.remaining_balls.remove(leftmost)
        self.remaining_balls.remove(secondleftmost)
        self.fill_last()
    else:
        raise ValueError("Incorrect Starting Configuration")

That fill_last() is doing the work of moving up a row, and adding the necessary ball in the third position. You’ll see that this function will get called every time a ball is added to the rack, but only for the topmost ball necessary—the top of whatever triangle exists.

def fill_last(self):
    current_last_length = self.rack_size+1-len(self.rows)
    diff = self.operation(self.rows[-1].elements[0], self.rows[-1].elements[1])
    if diff in self.remaining_balls:
        self.rows.append(row(current_last_length,diff))
        self.remaining_balls.remove(diff)
    else:
        raise ValueError("Impossible Rack")

So now, we just keep adding balls to the bottom row, and then filling out the rows above, and finishing off with fill_last():

def add_ball(self, value):
    if value in self.remaining_balls:
        self.rows[0].append(value)
        self.remaining_balls.remove(value)
        for i in range(1,len(self.rows[0].elements)-1):
            diff = self.operation(self.rows[i-1].elements[-1],self.rows[i-1].elements[-2])
            if diff in self.remaining_balls:
                self.rows[i].append(diff)
                self.remaining_balls.remove(diff)
            else:
                raise ValueError("Impossible Rack")
        self.fill_last()
        return self.remaining_balls
    else:
        raise ValueError("Impossible Rack")

This is all well and good, and does yield a solution for the original problem if you iterate through all the possibilities, but I wanted to see the evolution of these solutions, i.e. as you add balls, how many racks are floating around. First, this code does the iteration:

def solve_puzzle(puzzle_size, operation=delta):
    if puzzle_size<3 or puzzle_size>7:
        raise ValueError("Change your puzzle size")
    rack_list = []
    number_of_balls = ((puzzle_size*(puzzle_size+1))/2)+1
    for i in range(1,number_of_balls):
        for j in [x for x in range(1,number_of_balls) if x != i]:
            try:
                rack_list.append(rack(i, j, operation, puzzle_size))
            except ValueError:
                pass

And then I got a wild hair and wrote this atrocity:

new_rack = None
some_racks = None
def compute_subracks(new_rack):
    some_racks = []
    for ball in new_rack.remaining_balls:
        try:
            sub_rack = copy.deepcopy(new_rack)
            sub_rack.add_ball(ball)
            some_racks.append(sub_rack)
        except ValueError:
            pass
    return some_racks

def print_rack_tree(root_rack, puzzle_size):
    possibility_counts = {3:0,4:0,5:0,6:0,7:0}
    for r in compute_subracks(root_rack):
        possibility_counts[3]+=1
        if puzzle_size>=4:
            for s in compute_subracks(r):
                possibility_counts[4]+=1
                if puzzle_size>=5:
                    for k in compute_subracks(s):
                        possibility_counts[5]+=1
                        if puzzle_size>=6:
                            for m in compute_subracks(k):
                                possibility_counts[6]+=1
                                if puzzle_size>=7:
                                    for n in compute_subracks(m):
                                        possibility_counts[7]+=1
                                        n.print_rack()
                                else:
                                    m.print_rack()
                        else:
                            k.print_rack()
                else:
                    s.print_rack()
        else:
            r.print_rack()
    return possibility_counts

which, OMG, I cannot, for the life of me, refactor to be recursive. I have tried like three times and keep messing it up. It clearly should be recursive, but it’s above my pay-grade.

So now, a little word on these counts:

global_counts = {2:0,3:0,4:0,5:0,6:0,7:0}
for r in rack_list:
    global_counts[2]+=1
    counts = print_rack_tree(r,puzzle_size)
    for val in counts:
        global_counts[val]+=counts[val]
return global_counts

Which is going to yield the sequence leading up to a solution. Without further ado:

print solve_puzzle(5)

returns

row 4:         [5]
row 3:       [4, 9]
row 2:     [7, 11, 2]
row 1:   [8, 1, 12, 10]
row 0: [6, 14, 15, 3, 13]
Remaining: []
row 4:         [5]
row 3:       [9, 4]
row 2:     [2, 11, 7]
row 1:   [10, 12, 1, 8]
row 0: [13, 3, 15, 14, 6]
Remaining: []
{2: 196, 3: 1574, 4: 1734, 5: 2}

I know whatcha thinking, I was thinking it too. What about the racks for larger numbers(the astute reader will have noticed how often I left these functions to take n). Let’s look at 6:

{2: 400, 3: 5516, 4: 25994, 5: 3182, 6: 0}

No solution! Damn! But 3182 racks get to the level 5 before they fail at 6. Take note that the number of balls is larger, so there are more degrees of freedom here. Here are the first four it spits out, it’d be curious to see if there is some common structure, but there are a lot to look at.

row 4:         [4]
row 3:       [5, 9]
row 2:     [7, 12, 3]
row 1:   [13, 6, 18, 15]
row 0: [1, 14, 20, 2, 17]
Remaining: [8, 10, 11, 16, 19, 21]
row 4:         [5]
row 3:       [4, 9]
row 2:     [7, 11, 2]
row 1:   [13, 6, 17, 15]
row 0: [1, 14, 20, 3, 18]
Remaining: [8, 10, 12, 16, 19, 21]
row 4:         [4]
row 3:       [5, 9]
row 2:     [6, 11, 2]
row 1:   [13, 7, 18, 16]
row 0: [1, 14, 21, 3, 19]
Remaining: [8, 10, 12, 15, 17, 20]
row 4:         [6]
row 3:       [2, 8]
row 2:     [9, 11, 3]
row 1:   [14, 5, 16, 13]
row 0: [1, 15, 20, 4, 17]
Remaining: [7, 10, 12, 18, 19, 21]

Seven is similarly dissapointing, and so I’m just guessing that 5 is the upper bound for a solution(I’m going to try to prove this at some point, but I’m a bit lazy and uninspired).

Ok, so what about 4?

row 3:       [3]
row 2:     [4, 7]
row 1:   [5, 9, 2]
row 0: [6, 1, 10, 8]
Remaining: []
row 3:       [3]
row 2:     [5, 2]
row 1:   [4, 9, 7]
row 0: [6, 10, 1, 8]
Remaining: []
row 3:       [3]
row 2:     [2, 5]
row 1:   [7, 9, 4]
row 0: [8, 1, 10, 6]
Remaining: []
row 3:       [4]
row 2:     [2, 6]
row 1:   [5, 7, 1]
row 0: [8, 3, 10, 9]
Remaining: []
row 3:       [3]
row 2:     [7, 4]
row 1:   [2, 9, 5]
row 0: [8, 10, 1, 6]
Remaining: []
row 3:       [4]
row 2:     [5, 1]
row 1:   [2, 7, 6]
row 0: [8, 10, 3, 9]
Remaining: []
row 3:       [4]
row 2:     [1, 5]
row 1:   [6, 7, 2]
row 0: [9, 3, 10, 8]
Remaining: []
row 3:       [4]
row 2:     [6, 2]
row 1:   [1, 7, 5]
row 0: [9, 10, 3, 8]
Remaining: []
{2: 80, 3: 262, 4: 8}

And 3(for completeness)?

row 2:     [3]
row 1:   [5, 2]
row 0: [1, 6, 4]
Remaining: []
row 2:     [3]
row 1:   [4, 1]
row 0: [2, 6, 5]
Remaining: []
row 2:     [2]
row 1:   [3, 5]
row 0: [4, 1, 6]
Remaining: []
row 2:     [3]
row 1:   [2, 5]
row 0: [4, 6, 1]
Remaining: []
row 2:     [1]
row 1:   [3, 4]
row 0: [5, 2, 6]
Remaining: []
row 2:     [3]
row 1:   [1, 4]
row 0: [5, 6, 2]
Remaining: []
row 2:     [2]
row 1:   [5, 3]
row 0: [6, 1, 4]
Remaining: []
row 2:     [1]
row 1:   [4, 3]
row 0: [6, 2, 5]
Remaining: []
{2: 24, 3: 8}

You may think at this point “Can we stop talking about this?”, or you may think “what about if it was add instead of difference?”

Nope. Not even a little. With enough balls for a rack of size 6, there are only 7 additive 4-racks! Wild!

All of the above discussion and code in this gist.

Letter-place frequencies

This one also originated from Wolfram Alpha Can't, and asks about the most common letter, by place, by length of word.

I just love this question so much. The solution doesn’t take much work, just a little bashy loop:

for i in `seq 1 15`; do
    for j in `seq 1 $i`; do
              { cat /usr/share/dict/words | awk "length($1) == $i { print tolower($1) }" | sort | uniq | awk -v j="$j" '{print tolower(substr($0,j,1))}' | sort | uniq -c | awk '{print $2, $1}' | sort -nk2 | tail -1 | awk '{print $1}'; } | tr "\n " " " | tr -d '[:space:]';
    done
    printf "\n"
done

which has the added fun of making “words” out of the answers.

ao
aae
saae
saaee
sariee
saraiee
sareliee
sereeliee
pereoiaiie
pereooatiie
pnreoooatiie
pnteooiiatiis
pnteoooiianiss
pneeooooiianiss

I got really excited and wanted to look at a slightly more dramatic presentation, so I made this which I think is way cool! Check out those patterns!!!

The 538-multiplication problem

This is one of the 538 Puzzlers, and while I like the official solutions, I also kinda like my work on it:

10k sims:
random_placement: 2321,
mean_method: 1083,
thresh: (4, 4, 3, 1072),
cond_thresh: (4, 4, 3, 1072),
perfect: (4, 4, 3, 875)

And some cool pics.

End matter

All for now, but there will always be more. Let me know if you find a puzzle that I might like, or you have some input on the above.

Puzzles was originally published by Bryan Bischof at Hyperbolicity on May 19, 2017.

Random Partition

2017-05-14T12:08:50+00:00

I’ve been curious about generating a random partition of an integer.

RanCom

I have been thinking a lot about this somewhat randomly. In particular, I’ve been trying to think of how to select things randomly in various incarnations. This one wasn’t obvious other than the naive solution of

Generate all partitions
Select one at random

After some Google, I found this, which after a little thinking I was able to understand. The answer I used comes from this book—which I am now dangerously interested in reading. It becomes:

import random

def rancom(n,k):
    if k>1:
        a = sorted(random.sample(list(range(1,n+k)), n+k-1)[:k-1])
        return [a[0]-1]+[x[1]-x[0]-1 for x in zip(a,a[1:])]+[n+k-1-a[-1]]
    else:
        return [n]

Notice that this is for arbitrary k, but in our case, k=n

Nice and fast. I think it’d be fun to look at the balls-and-boxes algorithm that leads to this solution.

There’s also the fun question of what is an average partition for n:

def avg_part(n, iterations=500):
    parts = []
    for i in range(iterations):
        parts.append(sorted(rancom(n,n), reverse=True))
    for j in range(n):
        print int(round(sum([x[j] for x in parts])/len(parts)))

[2, 0]
[2, 1, 0]
[3, 1, 0, 0]
[3, 1, 1, 0, 0]
[3, 2, 1, 0, 0, 0]
[3, 2, 1, 1, 0, 0, 0]
[3, 2, 1, 1, 0, 0, 0, 0]
[4, 2, 1, 1, 1, 0, 0, 0, 0]
[4, 2, 2, 1, 1, 0, 0, 0, 0, 0]
[4, 2, 2, 1, 1, 1, 0, 0, 0, 0, 0]
[4, 3, 2, 1, 1, 1, 0, 0, 0, 0, 0, 0]
[4, 3, 2, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
[4, 3, 2, 2, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]
[4, 3, 2, 2, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]
[4, 3, 2, 2, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
[4, 3, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
[5, 3, 2, 2, 2, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[5, 3, 2, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[5, 3, 3, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[5, 3, 3, 2, 2, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[5, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[5, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[5, 4, 3, 2, 2, 2, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Random Partition was originally published by Bryan Bischof at Hyperbolicity on May 14, 2017.

A Horsey Race

2017-05-06T12:08:50+00:00

I’ve been playing with some puzzles lately(slightly egged on by Jeff).

Some recent puzzles

I’ve thought about some of the recent puzzles, even automating some and making some fun visualizations.

Also spent a little time writing some puzzles myself, and some out of left field questions.

However, for the most recent 538 puzzle, they suggested making an animation, so I thought I’d spent a few hours making a d3 vis for fun. Here’s the puzzle.

Horseys

The setup is relatively straightforward; using the list of probabilities, generate 20 sequences of steps, continuing to sum them until one exceeds 200. Here’s the little function—keep in mind that I don’t know Javascript:

function run_race(race_length){
  horsey_probabilities = [.52,.54,.56,.58,.60,.62,.64,.66,.68,.70,.72,.74,.76,.78,.80,.82,.84,.86,.88,.90].reverse()
  horsey_paths = [[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]]
  winning_position = 0
  while (winning_position < race_length) {
    new_steps = horsey_probabilities.map(function(d){return rand_step(d)})
    horsey_paths.map(function (list, idx) {
      return list.push(new_steps[idx]);});
    winning_position = Math.max.apply(Math, horsey_paths.map(function(d){return d.reduce((a, b) => a + b, 0)}))
//   console.log(winning_position);
  }
  return horsey_paths
}

The ‘hardest’ part was that I wanted to display the data as a bump chart of who was winning the race at each step. So I had to compute the ‘position’ at each step. Consider the following problem:

Given a square matrix of numbers of integers, a list of rows, convert the integers to the columnar rank of each entry.

For example:

[
  [1,2,3],
  [3,2,1],
  [5,1,0],
]

becomes

[
  [1,1,3],
  [2,3,2],
  [3,1,1],
]

It’s a little strange, but mostly involves changing the direction of the lists, and then computing the ranks; even in Python gave me pause:

a = [5,2,4,3,0]
[z[1] for z in
 sorted([(y[0],j+1) for j,y in
         enumerate(sorted([(i+1,x) for i,x in
                           enumerate(a)
                          ], key=lambda x: x[1]))
        ], key=lambda x: x[0])
]

So you can imagine it was a little awkard in JS, but here ya go:

function matrix_transpose(matrix){
  return matrix[0].map(function(col, i) {
    return matrix.map(function(row) {
        return row[i];
    });
});
}

function convert_list_to_ranks(list_of_sums){
  output = list_of_sums.map(function (i,idx){return ([idx+1, i]);})
    .sort(function(a,b) {return b[1] - a[1];})
    .map(function (p,idy){return ([p[0], idy+1])})
    .sort(function(a,b) {return a[0] - b[0];})
    .map(function (x){return x[1]})
  return output
}

Without further ado, here’s the vis.

A Horsey Race was originally published by Bryan Bischof at Hyperbolicity on May 06, 2017.

Joining Blue Bottle

2016-07-30T12:08:50+00:00

After several months of extremely casual job-searching, something clicked. And in some sense, that which clicked, was that confirmation-bias-makes-destiny’s-work-easy job.

The lead-up

I found this opportunity on one of the job forums. I was cruising and looking for anything that jumped out. I was really taken aback when I saw that Blue Bottle(BB) was hiring a data engineer. As most know, I’m really interested in coffee. I’ve been interested in the domain for more than five years now, and in that time, I’ve really gone out of my way to understand the industry as an outsider.

While I know a lot of people are “into” coffee, I think that this undersells my passion. I’ve genuinely wanted to work with coffee data for some time, but unfortunately, I haven’t really found any datasets that got me charged. Last year, I tried to contact Cropster, but I struggled to find a reasonable vector. Some other things I’ve thought about is collecting data on my own coffee consumption, but obviously this isn’t exactly huge. (I do intend on building some visualizations of my coffee drinking throughout the past couple years, as per my coffee bags.)

I shot off a cover letter to BB and was a little hesitant. My biggest concern was truly that the job seemed very focused on ETL and maybe not enough ML/Eng.

In comminique

They got back to me, we had the standard phone calls(the one with their digital lead was especially fun), and I got invited on site last monday. Due to a string of less-than-thrilling on-sites this year(I’m planning a write-up of those), I was quite nervous.

To skip ahead a bit, I told my friend Jeff monday night that “if I don’t get this job, I don’t know anything about getting jobs.”

The reason for the above statement was that I essentially killed my interviews. I felt confident in my answers, extremely personable that day, exuding excitement, and genuinely like I had skills to contribute. I got a little side-tracked in one interview, but otherwise I really felt on.

The next day I got an offer. And in-fact, signed it the same day.

Future perfect

I really can’t speak highly enough about the opportunity to be doing data-engineering and data-science for a coffee company. I feel like a dream has come true(sorry, I know this is corny).

While this means I’m leaving IBM a little earlier than I would have liked, I think it’s a good time for that anyhow. Despite there being a lot left to learn there, I think the move will dramatically benefit my mental health. One of the best things about Blue Bottle: it’s still in the East Bay!!!

Joining Blue Bottle was originally published by Bryan Bischof at Hyperbolicity on July 30, 2016.

Code for America App

2016-07-23T12:08:50+00:00

The notion of Code for America is very attractive to me. In particular, the allure of working directly with a local government, is one of the ideal jobs I could ask for. While there would be some huge sacrifices to taking this position(not developing certain skills, forfeiting access to certain computational infrastructures, and of course, the money aspect), I am still interested. So I decided to apply.

Data Challenge

The coding challenge was very very easy:

Given the following `csv`, compute the number of violations by type, and the first and last violation in each set.

And they provide a little data file with columns:

violation_id,inspection_id,violation_category,violation_date,violation_date_closed,violation_type

so the actual problem just requires a groupby, count, and min/max. However, the challenge asks specifically for you to present this output. Whence I decided to just use js/D3 for the whole challenge.

I had to remember how to use nest():

d3.nest().key(function(d) {return d.violation_type}).entries(data)

and then I did the fun thing of deciding how to draw this data. The data sets were so small, and so limited in time-scale, but I like the idea of something histogram-y. Since the number of total points was tiny, and one-dimensional, I remembered that beeswarms could be usable. So I went with that.

Here’s the live version.

An Alternative

The reality is, while the beeswarm is fun and kinda cool, I have to admit that it might just be better to use a calendar visualization. Because it’s all a single year I can use the exploded version of the calendar that takes up a bit more space and really highlights the delineation between the months.

Here’s the live version.

Code for America App was originally published by Bryan Bischof at Hyperbolicity on July 23, 2016.