Category Archives: Dev

A crumbling railroad bridge over snow, the uprights are worn and look ready to give way.

Zero to One to Crumbling

It’s a golden age of convenience and ease! The speed that we can get something usable up, working and deployed these days is incredible. The sheer amount of good engineering infrastructure we can take for granted is astounding – and that’s good.

In Deb Chachra‘s How Infrastructure Works she has a tricky definition of infrastructure that really works. Roughly it’s that “infrastructure what you can take for granted”. It’s the stuff you don’t have to think about. In my neighborhood, that’s a lot! I don’t think about water, light, heat, electricity, or food supply. At work, I don’t really think much about compute availability or disk availability – though I do have to design systems that use these mindfully at scale.

In software, there’s other kinds of infrastructure. Do you have to manually allocate memory? Do you have to destroy objects and free up allocated memory when you are done with it or does a garbage collector handle that for you? Do you have to optimize your control flow and loops or does an optimizer do an incredibly smart job of that for you? The more that there is good software infrastructure, the more you can spend time on the biggest difference maker for you – handling the business domain problems that you want to solve.

Old Man Story Time:

For my first job in NYC, I was given a C# code test. This was difficult for me, as I had no copy of VisualStudio.net available to me. Also, I didn’t know C# and figured it can’t be harder than C. I had to look up online manuals of C#, write the code in Notepad, and use csc.exe to test if it worked. A lot of what the test demonstrated was that I could write a for loop and that I was bull headed enough to push through obstacles and find out how to get something done. Now I write fewer for loops.

The last repository push I made I didn’t bother locally cloning – I was able to use the web based editor in GitLab to branch, make a change with auto-suggests, then commit, push and create a merge request. Unbelievably easy and cool.

But Jim Nielsen‘s Zero to Unmaintainable raises a good point:

There is such a focus on how quickly you can get going, but so little focus on how you maintain what you just created.

The developers I work with are so bright and smart and full of ideas. And they can get smarter, more reactive, better designed systems than I can think of up and running so quickly it’s astounding. But I see them frustrated, because they are held up by different things than I was. I used to get frustrated by the failures of the technology or the ability to even get simple things connected. They have best practices built into the new tools. But they are now frustrated by the organizational demands associated with new tooling.

Who will support this new application? How will we keep it updated? How will we secure it? When the APIs we interact with change, is this application well enough documented that someone else can fix the problem or do we need to take you off a project to fix it?

Part of my job is to help identify these issues early and turn them into infrastructure so that they can concentrate on the business problem and solutions to them – rather than these concerns. Instead of frustrated developers working out non-functional requirements, it’s better when we turn them into things they can take for granted.

This means working out commonalities across solutions we build for clients. Deployment’s a big one to get right once and then not talk about much. Same for version control. Same for dependency management and security scanning. Same for support after go-live. Same for maintenance. Same for self-service for clients.

The response to people being super fast to solve problem and come up with ideas isn’t to slow them down – it’s to solve the next bottlenecks.

These opinions are, of course, my own and published by me, not my employer.

Lessons Learned about AWS:ReInvent from AWS:ReInvent2023

I just came back from AWS:ReInvent 2023 and wanted to jot down some ideas I had about how to improve the experience next time.

First impressions – this conference was extremely well-run, very organized and FULL of helpful people eager to get you where you are going. Just like everything in Las Vegas, it felt like a thing designed to process you, a pipeline oriented around directing a sprayhose of humans to various areas. It’s people and crowd management at scale. That’s what it is and if you give up your individuality you can get whisked around and processed quickly. Swimming against the stream doesn’t work well.

Do’s and Don’ts:

  • Do register early and get flights and hotel rooms early. As a late registrant, I stayed in a non-conference hotel and had to do a lot of travel to get started where other attendees were.
  • Do try to keep most sessions within a venue. The transportation options are good, but walking in Las Vegas is a fool’s game. The conference centers are massive. The casinos are massive. It takes forever to get around. Even if you get on a shuttle, there is traffic. It’s difficult to move between venues.
  • Do reserve sessions seats early when they come out. Many sessions I wanted to attend were fully-reserved. This isn’t a big problem as there is plenty of space for walk-ins, but it means having to queue up for the walk-in space early, which limits your time and makes it hard to switch venues.
  • Do leave room for breakfast and lunch. It’s grueling focusing intently for long periods. Your brain needs energy for it. The AWS ReInvent folks provide meals for breakfast and lunch, but it’s easy to miss them if you have a session at that time. This isn’t such a big deal, except the venues are massive, so it takes a long time to get out and get to somewhere else that has decent food. The food at the venues was much healthier and better than a lot of the easy-to-get food in the casinos etc.
  • Don’t get too hyped about RePlay unless that’s really your jam. I’m sad I didn’t get there in time for the Linda Lindas, but conference wrap parties in general seem like getting on a bus, getting a few beverages, watching the band, getting back on a bus. Our bus got stuck behind tons of other busses on the road and we eventually walked to the venue. It’s a logistics nightmare to try to get 65K people in and out of a single concert in a reasonable timeframe. I had a lot more fun doing side events with my co-workers like checking out the Sphere and going to OmegaMart.
  • Don’t go to “Builder’s Sessions”. Your mileage may vary, but I found these uniformly unhelpful. The session is a very brief talk, then following a tutorial at a rapid pace. There is very little context. The session names seemed interesting, but generally they were not educational – they were just a way to step quickly through a pre-configured tutorial. There was no explanation of WHY to do anything – many of the instructions were “copy paste this text”. At the end of any of these sessions, there was no time to ask questions and I don’t think I got much from them. I don’t think anyone who attended one added any new skill.
  • Do go to big talks in big rooms. The best talks get the big rooms. Figure out capacity and try to attend these. The best talk I attended, “Building Observability to Increase Resiliency” was in a big room, and I loved it.
  • Do meet with product managers for products you use. You get a lot of information and they can speak about things that aren’t published or take your specific nuanced feedback about usage. I found this super useful. I got to talk about what would specifically impact my team and our way of working.
  • Do watch the keynotes via stream in your hotel room. They don’t hand out any special goodies. The hotel is comfy. The editing is good.
  • Do register for new sessions after they keynote. They unlock new sessions related to the just announced stuff. Most of the just announced stuff didn’t really thrill me, but there is a helpful filter in the catalog for “just launched” sessions.

Most of the sessions I attended really were laser focused on the AWS product, which, you know, fair enough. It’s not a generalist conference. But my favorite session took great lessons about a subject and then applied them to how to use AWS products.

Every Frame a Wallpaper

Years ago, Tony Zhou and Taylor Ramos made 28 video essays about film form called Every Frame a Painting and it’s incredible in teaching outsiders a whole new way to think about the art of film. The title is perfect. The content is just stunning, simple ways to look at masters of a form at work.

Lots of folks are known for one-shot takes, but this shows how Spielberg sneaks in gorgeous “oners” that do work without calling attention to themselves.

This essay on Fincher is great, but I love the little golden nugget about how spacing shows the evolving relationship between Mills and Somerset

That title always struck me. Every Frame a Painting. That’s gotta be a bar filmakers strive for. Some make it.

Some movies are just so damn beautiful. Just gorgeous.

Like Across the Spider-Verse. Yowza!

Like Sita Sings the Blues! Beautiful.

Like The Fountain

Like the one that you like that isn’t my cup of tea.

Might be nice to see an image from it, right there behind all of your terminals and windows and such, set as your wallpaper. If every frame’s a painting, set a random one as your wallpaper whenever I like it.

So here’s the plan. I want it. So I made it for me. You can have it. But here’s the terms of the deal. I made it for me, so if it doesn’t work for you, you have to make it work for you. If it causes you problems, those are not my problems. If you don’t agree, this isn’t for you.

This will take as an input a movie file, anything that ffmpeg can deal with. You’ll need to install ffmpeg – look on the official site for instructions.

By default, it won’t use the first 5 or last 10 minutes since that’s often the credits. But you can override this.

We’ll find out how many frames are in that remaining part of the movie.

We’ll pick one randomly and extract it from the movie.

Then we’ll set it as your wallpaper. Nice!

Want to change this often? Set up a cron job!

Pulling a single frame out the middle of a movie is CPU intense, so you probably want to use nice in your cron job so it doesn’t interfere with the rest of your work.

Here’s the code, save this in a file called every_frame_a_wallpaper.zsh and then chmod u+x every_frame_wallpaper.zsh

#! /bin/zsh
# This is a pretty processor intensive set of tasks! You should probably nice this script
# as in call it with nice -n 10 "every_frame_a_wallpaper.zsh /path/to/video.mkv"

SCRIPT_NAME=$(basename "$0")

# I like a nice log file for my cron jobs
function LOG() {
  echo -e "$(date --iso-8601=seconds): [$SCRIPT_NAME] :  $1"
}

# set up some options
local begin_skip_minutes=5
local end_skip_minutes=10
local wallpaper="$HOME/Pictures/wallpaper.png"
local usage=(
	"$SCRIPT_NAME [-h|--help]"
	"$SCRIPT_NAME [-b|--begin_skip_minutes] [-e|--end_skip_minutes] [<video file path>]"
	"Extract a single random frame from a movie and set it as wallpaper"
	"By default, skips 5 minutes from the beginning and 10 from the end, but this is overridable"

)

# the docs suck on zparseopts so let this be a reference for next time
# -D pulls parsed flags out of $@
# -F fails if we find a flag that wasn't defined
# -K allows us to set default values without zparseopts overwriting them
# Remember that the first dash is automatically handled, so long options are -opt, not --opt
zparseopts -D -F -K -- \
	{h,-help}=flag_help \
	{b,-begin_skip_minutes}:=begin_skip_minutes \
	{e,-end_skip_minutes}:=end_skip_minutes \
	|| return 1

[[ -z "$flag_help" ]] || {print -l $usage && return }
if [[ -z "$@" ]] {
   print -l "A video file path is required"
   print -l $usage && return

} else {
   MOVIE="$@"
}

if [[ $DISPLAY ]]
then
  LOG "interactively running, not in cron"
else
  LOG "Not running interactively, time to export the session's environment for cron"
  export $(xargs -0 -a "/proc/$(pgrep gnome-session -n -U $UID)/environ") 2>/dev/null
fi

LOG "skipping $begin_skip_minutes[-1] minutes from the beginning"
LOG "skipping $end_skip_minutes[-1] minutes from the end"
LOG "outputting the wallpaper to $wallpaper"
LOG "using file $MOVIE"

LOG "Let's get a frame from ${MOVIE}";


LOG "What's the duration of the movie?"
DURATION=$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 \
  $MOVIE);
DURATION=$(printf '%.0f' $DURATION);

LOG "Duration looks like ${DURATION} seconds";

LOG "what's the frame rate?"

FRAMERATE=$(ffprobe -v error -select_streams v:0 \
  -show_entries \
  stream=r_frame_rate \
  -print_format default=nokey=1:noprint_wrappers=1 $MOVIE)




FRAMERATE=$(bc -l <<< "$FRAMERATE");

FRAMERATE=$(printf "%.0f" $FRAMERATE);

LOG "Looks like it's roughly $FRAMERATE"

FRAMECOUNT=$(bc -l <<< "${FRAMERATE} * ${DURATION}");
FRAMECOUNT=$(printf '%.0f' $FRAMECOUNT)
LOG "So the frame count should be ${FRAMECOUNT}";


SKIP_MINUTES=$begin_skip_minutes[-1]
SKIP_CREDITS_MINUTES=$end_skip_minutes[-1]

LOG "We want to skip $SKIP_MINUTES from the beginning and $SKIP_CREDITS_MINUTES from the end".

SKIP_BEGINNING_FRAMES=$(bc -l <<< "${FRAMERATE} * $SKIP_MINUTES * 60");
LOG "So $SKIP_MINUTES * 60 seconds * $FRAMERATE frames per second = $SKIP_BEGINNING_FRAMES frames to skip from the beginning."
SKIP_ENDING_FRAMES=$(bc -l <<< "${FRAMERATE} * $SKIP_CREDITS_MINUTES * 60");
LOG "So $SKIP_CREDITS_MINUTES * 60 seconds * $FRAMERATE frames per second = $SKIP_ENDING_FRAMES frames to skip from the ending."

USEABLE_FRAMES=$(bc -l <<< "$FRAMECOUNT - $SKIP_BEGINNING_FRAMES - $SKIP_ENDING_FRAMES");
UPPER_FRAME=$(bc -l <<<"$FRAMECOUNT - $SKIP_ENDING_FRAMES")
LOG "That leaves us with ${USEABLE_FRAMES} usable frames between $SKIP_BEGINNING_FRAMES and $UPPER_FRAME";

FRAME_NUMBER=$(shuf -i $SKIP_BEGINNING_FRAMES-$UPPER_FRAME -n 1)
LOG "Extract the random frame ${FRAME_NUMBER} to ${wallpaper}";
LOG "This takes a few minutes for large files.";
ffmpeg \
  -loglevel error \
  -hide_banner \
  -i $MOVIE \
  -vf "select=eq(n\,${FRAME_NUMBER})" \
  -vframes 1 \
  -y \
  $wallpaper


WALLPAPER_PATH="file://$(readlink -f $wallpaper)"
LOG "Set the out file as light and dark wallpaper - using ${WALLPAPER_PATH}";
gsettings set org.gnome.desktop.background picture-uri-dark "${WALLPAPER_PATH}";
gsettings set org.gnome.desktop.background picture-uri "${WALLPAPER_PATH}";

In my crontab I call it like this:

# generate a neat new background every morning
0 4 * * * nice -n 10 ~/crons/every_frame_a_wallpaper.zsh -b 5 -e 12 /home/mk/Videos/Movies/Spider-Man_Across_the_Spider-Verse.mkv >> ~/.logs/every_frame_a_wallpaper/`date +"\%F"`-run.log 2>&1

A little development environment in vim and tmux

I’ve been doing a little project and took a moment to get a bit better at using tmux.

Every time I go into this project I set up some splits. A main window where I’ll edit files using vim, then I split a pane off to run the code or test suite on every save. Another split where I pip install after any changes to my requirements.txt file.

Since I do the same thing repeatedly I was pretty sure tmux has a way to set this up so I don’t need to do it by hand. I tried using tmux session saving plugins, but they are too much for what I need right now.

Turns out tmux is incredibly easy to script. This gist is very long and very informative on how to split windows in tmux and covered everything I needed.

#! /bin/sh

# split -h horizontally to take up 30% of the width to run my __main__.py file on every save of a python file
# this is -d detached so that focus remains on the main window
tmux splitw -d -h -p 30 'ls *.py | entr -c env/bin/python . ./goodreads_library_export.csv data.csv ~/books'
# split my second pane vertically with 20% for rerunning pip installs on save of requirements.txt
tmux splitw -d -t 2 -p 20 'ls requirements.txt | entr -c env/bin/pip install -r requirements.txt'
# create a little detached shell just in case I need to try something
tmux splitw -d -t 3
# open up the python files in tabs in my main pane
vim -p *.py 

entr is a great little tool I like for monitoring for file changes and running a command in response.

a quick development environment for a project

Grepping a process output

I’m just noting this down because I had to do a lot of reading to get this right. Now that I’m actually using log libraries in a good way for my scripts, I want to dial up and down the log level easily – but setting the loglevel on my overall script to debug makes EVERYTHING output, and I don’t really need every library’s output. I’m sure there’s a better way to control the log level of a specific logger for my script from the command line, but this works for me in a quick and dirty way.

env LOGLEVEL=DEBUG ./myscript.py |& grep my_logger_name

I set the log level for just this run, then use the |& to capture both stdout and stderr, then pass that to grep.

Automated export of your goodreads library

Goodreads used to have an API but they stopped giving access and it looks like they are shutting it down. A real garbage move.

I like to be able to use my data that I put in so I wrote a script to automatically download my data regularly. Then I can do stuff like check to see if books I want are in the library or keep my own list or analytics, etc.

Here’s the python script to export your good reads library, hope it helps you. I’ll put it in the public domain.

Updated to add: I got tired of dealing with places that do garbage moves. I left GoodReads for BookWyrm and it’s better.

Week 2206

Politics

I donated money to the Alex Morse campaign, a progressive candidate who’s trying to unseat Richard Neal, a greedhead Democrat. That happened earlier, but recently it appears that there was a sex scandal accusation against Morse. He’s accused of having consensual sex with adult students at the university he teaches at that are not in his class and also messaging people he’s met on Tinder. Sexual harassment and consent are incredibly important, but weaponized accusations are exactly the sort of thing that conservatives have professed concern over. In Alex’s case, the investigation by the Intercept certainly makes it seem like people who want to work for Richard Neal have been manufacturing a scandal instead of uncovering one.

Other campaigns I’m looking closely at:

  • The State Slate – The great slate didn’t do great in 2018, but I still like these ideas and I’m willing to give again. These candidates are all good chances to flip a district and any campaigning they do is good for upballot races.
  • Donna Imam – an engineer who might be able to flip a texas district.
  • Dani Brzozowski

Family

The fam out at the Esopus Creek trail

We’ve been doing more hikes again. I’m trying to make sure the little monsters leave the house every day. We’ve been going out to the village a little bit as well. I haul the kids in our expandable wagon and we can eat at an outside restaurant called The Partition.

We’re getting an eensy bit more social (in safe and measured ways).

ZZ had an extended encounter with a nice lady named Alexa and her dog Chacho. They spent an hour hanging out and I can’t recall having a nicer meal in ages. Here’s pro tip – if you hang out with the children and amuse them while we have drinks and dinner I’m grabbing your bill!

Beer Club had a mini executive retreat when Ray showed up in Rhinebeck! We took the Ho’s to the FallingWater trail where I finally got to meet Finley! He loves Max and Zelda loves him.

The Scott’s dropped by! We took them out to Fallingwater as well, where Max and Ben got along really well and explored up the waterfall all the way to its source. Zelda is in love with Zoe and asks about her.

Max and Ben never usually play together, but for some reason this day was just perfect. Everyone got along famously.

DIY

Around the house, we’ve been struggling a little to knock out more projects. It just seemed like we lost steam. So we dug out the back yard next to our house and put in a bunch of marble rock chips over garden cloth. Now things are better looking and won’t require any weeding – instead of a dirt patch next to the house we have clean white stone which doesn’t need maintenance.

We cleaned out the trampoline, which had been under a mulberry tree, trying to become a mulberry jam strainer. Yecchh.

We spent a couple of meeting looking at adding solar panels to our roof – I really like it for a lot of reasons including my predilection for distributed systems over centralized ones. Sadly, the tradeoffs right now don’t seem worth it. Even with incredible financing and all sorts of incentives it would take forever to pay off the panels and require trimming trees.

This helped me feel like we can really start getting going again. I’m gonna finish that Patio!

Code and nerdery

Great news here! I’ve been thinking at work about ways to better handle and test documentation across multiple languages. The key here is to make sure that you can extract code samples from documentation and then push it out to a testable format.

I found mkcodes, an excellent tool for pulling code out of markdown documents. It worked great, but only for Python. I submitted a pull request extending it to work with multiple language code blocks and it was a real treat to work with Ryne Everett on getting this live. Which is to say, now it can handle java, dotnet, any other language you like that’s embedded in your docs. Expect more on this as I make progress building a docsite with eleventy.

I also type this on the linux laptop as I managed to resize partitions without destroying anything. I thought 80 gigs would be enough for my Ubuntu partition, but it seems to be growing and I had to give it a few hundred more gigabytes to grow.

Hosting better screencasts in WordPress

When I was writing up a note about pingplotter I figured I should show how it works because that’s much better for understanding it.

This gobbled up an incredible amount of time, so I’m writing it down so I never have to figure it out again.

There’s a few terminal screencast recorders out there, but I picked asciinema since it seemed simple, it recorder everything as text, not an image and it seemed a very light format to playback. These things are true, and the pain I felt is mostly unrelated to any failings of asciinema.

Asciinema is great. I can record a session locally, play it back to check locally and then upload it to their hosting to share. All pretty cool!

# record a session, but skip pauses over 1 second
asciinema rec -i 1 pingplotter.cast
asciinema: recording asciicast to pingplotter.cast
asciinema: press <ctrl-d> or type "exit" when you're done
# type a bunch of things to demo
# then ctrl-d to stop recording
asciinema: recording finished
asciinema: asciicast saved to pingplotter.cast

# time to check to see if it was any good
asciinema play pingplotter.cast
# the saved session plays back

# good enough!
asciinema upload pingplotter.cast 
# asciinema gives you the URL of the upload
# you can give it a description and make it public

Since this website is still powered by WordPress, I wanted to share it here. The Asciinema url didn’t embed automatically, so I figured I could either embed a script from Asciinema or add a plugin to do the embed code.

Both of these failed – somehow the javascript from Asciinema wasn’t working in my Firefox. It worked in Chrome, but that’s not good enough. I figured I should just host the cast myself! Why do I need to embed this tiny file or load javascript or install plugins to do this?

I found a neat utility to convert an asciicinema screencast to an animated SVG.

Listen, SVG and animated SVG is incredibly powerful, and if you don’t know, you should learn about it. Animated gifs are heavy, don’t resize, and are full of rendering artifacts. SVG are vector graphics, which mean they are infinitely scalable up or down crisply in a much smaller size than most gifs. Every modern mobile and desktop browser supports them.

The cool utility is svg-term. It lets you take an asciinema file and turn it into an animated svg.

It’s very nicely made, but is unfortunately distributed with npm, a package manager designed to ruin you. The instructions svg-term gives you to install it are incredibly dangerous unless you know to set up your npm install to not install packages with root privileges, WHICH FOR SOME REASON ISN’T THE DEFAULT OR EVEN POSSIBLE. I looked at the npm documentation for something like pip install --user, but it doesn’t exist. Instead, the solution is to redefine global installs to mean user local installs. I’m not kidding.

I don’t want to ever have to figure this out again either, so I added it to my jumpstart script so it will be ready for the next computer I use.

cat pingplotter.cast| svg-term --out ~/pingplotter.svg

This worked great! I went to upload it to my WordPress site to share with you. It did not work great!

WordPress very reasonably recognizes that while I enthusastically love SVG, it can be a security hole. SVG files can embed all sorts of foolishness like JavaScript and other loopholes to escalate privileges. So WordPress prevents them from being uploaded. I opted to allow them with a plugin that sanitizes SVG files to remove the dangerous stuff but allow the rich graphical goodness. This worked great!

I then decided to demo using tmux and glances, which meant I had to do a lot more typing in the demo, which expanded the size of the screencast. Poor svg-term began to crap out because I was exceeding the size of the javascript heap by feeding it a file larger than 1 mb. Fair enough, I don’t want to have a big file anyway!

The solution here is to script as much as possible ahead of time. The asciinema commandline takes a -c command argument which makes it execute a command as soon as it startes recording. Since tmux also lets you script up what you want it to do, I ended up with this.

asciinema rec -i 1 -c 'tmux new "pingplotter ; read" \; split-window "glances ; read" \; select-layout even-horizontal' pingplottertmux.cast

I’m telling asciinema to open up tmux and start recording. I’m telling tmux to start with pingplotter in a window, then open a new pane with glances and make the layout side-by-side. Each of those panes will close when the command they are running is done. Now I can just use ctrl-a <space> a few times to rotate the layout, ctrl-c to kill the panes, tmux exits and asciinema stops recording.

The file size ended up being small enough for svg-term and now you get this!

Anyhoo, black people are human beings with real lives and I don’t want my tax dollars to be used to murder them. Please, call your city representatives to urge them to shrink the police budget and use that money for social services that prevent crime and pain. If you can’t figure out your local representation, please learn how. If you won’t, please at least call your federal reps to ask them to stop giving the city police things like machine guns and tanks.

Pingplotter

I need to see if my remote connection to work is healthy and if my remote connection to video conferencing is suffering.

I wrote a little utility to handle that in the commandline. It’s dumb but it works. Let’s call it pingplotter. It’s in the public domain, go nuts.

It’s fine to just pop in a terminal you have open, but it plays really nice with tmux and glances (I use both).

I think I spent nearly as much time making these screencasts as I did writing the shell script. I should blog that too, if only so I don’t have to research that again.

One more thing to do after upgrading python

This is mainly a note-to-self to remind me for my next upgrade – but hope it helps you too. TL;DR – install python-dev tools, export old installs into a requirements file, import into new python site-packages.

I upgraded the python on my laptop from 3.6 to 3.7 so I could use dataclasses for a little project. Then I got on a plane to Zurich and planned to get some work done on the 9 hour flight. Unfortunately I spent most of that time wrangling my python install – little cli tools I like to use like black, glances, sphinx, cookiecutter, etc – none of them worked!

When you do a python pip install of a library, it puts the library in a directory called site-packages under lib/python3.<your-version>/site-packages/<your-package-name>. If the library has defined command line entry-points, you will find it has also installed a file under bin/<your-package-name>. If bin is in your PATH, it means you can just type something like cookiecutter and it will call a function in the library to do stuff for you! Here’s what one of those files looks like:

➜  ~ bat ~/.local/bin/cookiecutter
───────┬───────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /home/matt/.local/bin/cookiecutter
───────┼───────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/usr/bin/python3
   2   │
   3   │ # -*- coding: utf-8 -*-
   4   │ import re
   5   │ import sys
   6   │
   7   │ from cookiecutter.__main__ import main
   8   │
   9   │ if __name__ == "__main__":
  10   │     sys.argv[0] = re.sub(r"(-script\.pyw?|\.exe)?$", "", sys.argv[0])
  11   │     sys.exit(main())
───────┴───────────────────────────────────────────────────────────────────────────────────────────────
➜  ~

It tells the shell to execute it using /usr/bin/python3 – that’s a link to the latest version of python3 – in this case it’s now python3.7, while the library was installed under python3.6. So the command line now fails! Python3.7 doesn’t know about libs installed for 3.6 at all.

On the flight I realized I could either copy from the python3.6 site-lib to the python3.7 one or just switch python3 to point back to python3.6 to make things work. But once you have internet access, here’s the magic to get it all working.

First, install the python3.7 dev stuff so you can compile anything that needs compilation.

➜  ~ sudo apt-get install python3.7-dev

Then let’s export everything we used to use under 3.6 and then reinstall it under 3.7.

➜  ~ python3.6 -m pip freeze > old_requirements.txt
➜  ~ python3.7 -m pip install --user -r old_requirements.txt