Little Jinja Tricks

 These are a bunch of Jinja-related things I searched for once and might need again...

Stripping newline characters in a for loop

If you're iterating over a list and want to the output presented as:

ItemA ItemB ItemC ...

Instead of:

ItemA
ItemB
ItemC
...

You can do: {% for item in list -%} {{ item }} {%- endfor %}
Note the '-' in the for/endfor block.

Ensuring a list with one item is treated as a single-item list

Iterating over a list with exactly one string element in Jinja will generally lead to a situation where the element is parsed as i n d i v i d u a l  c h a r a c t e r s.  To avoid this, you can force the list to be interpreted as a list and specify the delimeter:

{% set list1 = my_list.split(',') %}
{% for item in list1 %}
{{ item }}
{% endfor %}

How to get elements from a dictionary of dictionaries

Sometimes your data is not stored conveniently in a simple list or dictionary.  Sometimes the values in your dictionary are actually dictionaries themselves.  In this case, you can use the dict.items() method:

{% for key, value in my_dictionary.items() %}
{{ value.sub_key }}
{% endfor %}

Other things...

More goes here.

Renaming Linux network interfaces via udev rules

I had this issue recently when I set up a new server with a bunch of built-in ports.  The default interface assignments when I performed my initial install of Ubuntu on the device made perfect sense (e.g., eno1 was "port 1" from the server guide, etc.), but after installing EVE-NG I ended up with ethX names that were spread all over the place.  This is what it took to get the names to make sense again, and note that this process makes use of the interface bus address, rather than MAC address, to ensure the name doesn't change.

Step 1:  Allow the kernel to rename network devices via udev

You should be able to do this in /etc/default/grub.  Edit the file, and make sure that GRUB_CMDLINE_LINUX includes net.ifnames=1.  After making the change (if necessary), run update-grub to update /boot/grub/grub.cfg.

Step 2:  Identify where the network interfaces appear on the PCI bus

Run the command lspci | grep net and check the output.  Here is what I see on my server:

root@superserver-01:~# lspci | grep net
65:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
65:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
65:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
65:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
b7:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
b7:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
b7:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
b7:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)

As you can see, there are two Ethernet controllers on-board, and each controller is attached to four physical interfaces.  Make note of the PCI address associated with each interface.

Step 3:  Create the udev rule

The udev rules are located in /etc/udev/rules.d.  Some OS'es will have a default file named 70-persistent-net.rules already there.  Whether you instance does or not, we will create another rule intended to preempt any changes the pre-installed rule will make.  So, create a new file named 60-persistent-net.rules.  For each network interface you wish to name, create a line like the following:

SUBSYSTEM=="net", ACTION=="add", KERNELS=="<pci_address>", NAME:="<if_name>"

In my case, I have the following:

root@superserver-01:~# cat /etc/udev/rules.d/60-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:65:00.0", NAME:="eth0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:65:00.1", NAME:="eth1"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:65:00.2", NAME:="eth2"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:65:00.3", NAME:="eth3"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:b7:00.0", NAME:="eth4"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:b7:00.1", NAME:="eth5"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:b7:00.2", NAME:="eth6"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:b7:00.3", NAME:="eth7"

Two things to note: First, the PCI address that you grabbed from lspci is prefixed by 0000: here.  Second, note that the name assignment includes the operator := rather than just =.  This ensures that the interface name will not be rewritten by subsequent rules.

Now you can save the changes and reboot.  When the server is up and running again, you should see that your interfaces are named how you want.

Good luck!

Linux Networking with Netplan: Examples, etc.

This is not meant to be a prose-y post.  It's really just a handful of examples to illustrate how to use netplan to implement host networking in Linux.  Remember to use sudo netplan generate to syntax-check your changes, and sudo netplan apply to implement them.

The netplan reference is located at https://netplan.io/reference/

Using DHCP

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth0:
      dhcp4: false
      dhcp6: false

Assigning a static IP address (with gateway, DNS, etc.)

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth0:
      dhcp4: false
      addresses: [192.168.100.118/24]
      gateway4: 192.168.100.1
      nameservers:
        search: [lab]
        addresses: [1.1.1.1, 9.9.9.9]

Don't wait for an interface on boot

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth0:
      optional: true
      dhcp4: false
      addresses: [192.168.100.118/24]
      gateway4: 192.168.100.1
      nameservers:
        search: [lab]
        addresses: [1.1.1.1, 9.9.9.9]

Creating a bond interface (also illustrates a static route)

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth1:
      dhcp4: false
    eth2:
      dhcp4: false
  bonds:
    bond0:
      interfaces: [eth1, eth2]
      addresses: [172.24.100.116/24]
      dhcp4: false
      dhcp6: false
      routes:
      - to: 172.16.0.0/12
        via: 172.24.100.1
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        transmit-hash-policy: layer3+4
        mii-monitor-interval: 100

Creating a VLAN-tagged interface

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth1:
      dhcp4: false
      dhcp6: false
  vlans:
    eth1.101:
      id: 101
      link: eth1
      addresses: [172.24.101.118/24]
      routes:
      - to: 172.16.0.0/12
        via: 172.24.101.1
    eth1.102:
      id: 102
      link: eth1
      addresses: [172.24.102.118/24]
      routes:
      - to: 172.16.0.0/12
        via: 172.24.102.1

Why all the Official Reporting on COVID is B.S.

An open letter to government and health care leaders across the country:

This is the story of why I’ve completely lost faith in the ability of the government and the health care system to handle the COVID-19 pandemic.  In short, you ignored logic, you ignored me and my family, and. you completely ignored the reality of the situation as it unfolded in January and February, leading to a completely failed response by March.

My family’s story begins back on Tuesday, the 25th of February.  That was the day my 12-year-old son developed a dry cough.  We didn’t think much of it at the time, it was already an early allergy season, but things were about to change quickly.  By Wednesday evening, my son was running a 101º fever.  It was too late to go to the pediatrician’s office, so we waited until Thursday morning.

We arrived at the pediatrician’s office shortly after they opened at 9am on Thursday.  By that point, my son’s cough was descending into his chest.  He was still running a 101º fever, and he was complaining of a sore throat.  The doctor tested him for strep, then she went through the COVID checklist.  No, he hadn’t been to China.  No, we didn’t know anyone who had recently been to China who had symptoms.  I didn’t understand why the doctor tested him for strep, the sore throat complaint I guess, and I should have pressed her harder on it.  She decided he had a respiratory infection, and she was confident it wasn’t COVID because he had no links to China.  So she sent us home to wait out his virus.

By mid-day Thursday, I started developing a dry cough myself.  Nothing severe, just a tickle at the back of the throat.  By Friday morning I was running a 101º fever too, and I could feel the cough creeping into my chest as well.  At this point my son and I both had chesty coughs and fevers.  Neither of us had body aches or headaches.  No runny noses or sinus pressure — this thing was just setting up shop in our chests.  It wasn’t strep, and it sure didn’t seem like flu (we’d already had two rounds of that in the house, in November and January).  I was starting to worry that this might be COVID.

My son and I suffered through the weekend.  My son’s fever broke on Sunday morning.  He was still coughing from his chest, and it was becoming a wet, phlegmy cough.  But he had missed so much school already this year that we sent him back  to school on Monday, the 2nd of March.

By evening that Monday, I was feeling miserable.  I was still running a fever and coughing terribly, but I also began to notice over the course of the afternoon that my breathing was becoming more labored.  I was now genuinely worried that my son and I had COVID, so I visited the local Righttime urgent care center.  I was seen promptly upon my arrival and I let the RNP know my concerns.  She read me the standard CDC script (have you been to China, etc.) and told me that clearly I didn’t have COVID because China.  She diagnosed me with an upper respiratory infection (again, I never had a runny nose or any sinus pressure) and said I was developing bronchitis.  She gave me a course of albuterol via nebuliser while I was in the office and sent me home with prescriptions for prednisone and an albuterol inhaler.  I was feeling much better after the nebuliser treatment.

On Tuesday evening, my son went to soccer practice.  He lasted about 15 minutes before he couldn’t continue.  He was coughing uncontrollably and complaining about pain in his chest, so my wife took him to the local Patient First health care clinic.  The doctor at Patient First diagnosed him with pneumonia in his left lung.  She sent him home with prescriptions for a Z-pack, augmentin, and an albuterol inhaler, along with guidance to see his pediatrician on Thursday.

For my part, Tuesday started out well, but by late afternoon I was feeling poor again.  My fever had finally broken, but the tightness in my chest was coming back.  And overnight Tuesday, I felt like I could feel “goo” moving in my chest any time I changed position in bed.  I was worried — both because I didn’t feel like I was getting better and because I was supposed to be going on a cruise with my wife on Thursday.  I needed to do something.

The morning of Wednesday, March 4, I made an appointment at the local Adventist Healthcare clinic.  (I thought I was going to the same place my son had visited the evening before, but I picked the wrong facility on the same block.)  While checking in, I specifically wrote on the check-in sheet that I thought I had COVID.  When I saw the doctor, she asked why I thought I had COVID.  I explained my symptoms — the fever, the cough, the difficulty breathing, and my son’s experience as well — and then she read me the standard CDC script.  No, I hadn’t been to China.  No, I don’t know anyone who had been to China who was exhibiting symptoms.  I only knew of my son, who was exhibiting COVID symptoms and had been diagnosed with pneumonia the night before, but had no connection to China.  The doctor did an X-ray of my chest and determined that I did not have pneumonia, but rather bronchoconstriction and possible fibrosis.  I received another duoneb treatment in the office, and the doctor sent me home with a prescription for a Z-pack and told me to double-up on my inhaler.  This time I challenged the doctor and asked why she didn’t think I had COVID based on my symptoms.  Her response was that based on CDC guidance, and specifically my lack of connection to China, that my symptoms did not indicate COVID.

The next day was March 5, the day we were supposed to leave for our cruise.  Once again, after the duoneb treatment, my chest felt much better.  Trying to be as objective as possible, I still felt like I had COVID, but now three different doctors had told both my son and me that we didn’t.  So, being told that I didn’t have COVID, and being fever-free since Tuesday, we decided to take our cruise.  Ironically, for all the warnings from the cruise line about enhanced screening measures, they let me on the ship without any questions.  They literally just looked at my passport to ensure I didn’t have a Chinese (and possibly Iranian) visa inside.  There was literally a family all wearing masks at the check-in station next to us, and they got on board too.  So much for enhanced screening.

By Saturday, March 7, I had started feeling dizzy while standing.  I wasn’t completely sure at the time if it was just a case of not getting my sea legs on the cruise or something more, but things seemed to be spinning more on this cruise than on previous ones.  That, and by the Saturday evening I was developing soreness under my left armpit.  That seemed odd.  By Sunday morning, I had soreness under both armpits, and the room was definitely spinning.  This was not normal, but I wasn’t about to let anyone know if I could avoid it.  I wasn’t going to be the guy who got another cruise ship quarantined.  I soldiered on, and finally by Wednesday, March 11, I started to feel normal again.  The soreness and the dizziness were gone, and so was the nagging cough.  Wednesday evening was the first time in almost two weeks where I could actually say I felt good.

Things were good for a couple of days when my wife started with a low-grade fever.  She had been having sinus issues for a couple of days, but we chalked that up to seasonal allergies as neither my son nor I had sinus issues with our infections.  But on Saturday, March 14, the fever started.  She ran a fever off-and-on until the following Friday, but the cough was getting worse the whole time.  Finally, on Friday, March 20th, she began a constant 101º fever.  Now with fever and cough, she was trending much like my son and I had just a couple of weeks before.  She made her visit to the local Righttime care center on Friday evening.  She let them know that she thought she might have COVID, and she got the usual battery of questions — still obsessed with China.  But they did test her for strep (negative), flu (negative), and RSV (negative).  She asked for a COVID test, but RIghttime did not have test kits available, nor did they feel comfortable/capable referring her to a facility that could test her.  They did give her a prescription for an albuterol inhaler and sent her home with directions to self-quarantine for 14 days.  Her discharge papers say, “Patient was instructed to self isolate for 14 days for possible concerns of community exposure to Covid.”  This was the first time anyone had acknowledged to a member of my family that we might have been exposed to Covid via means that didn’t involve direct contact with China.  And that "community" that exposed my wife to the disease?  That would be me, my son, and whomever infected my son.

Not satisfied with Righttime’s response, my wife contacted MedStar Montgomery Friday evening.  We called the ER at Montgomery General Hospital to ask about testing.  They told us they were only testing in-patients, and that we should contact one of MedStar’s PromptCare centers to request a test.  It was late enough Friday by this point that the PromptCare centers were closed.

So first thing Saturday morning, my wife contacted MedStar PromptCare.  She was told that she would have to do a virtual visit with a doctor first, and if the doctor thought a COVID test was necessary, the doctor would refer her for the test.  My wife had her virtual visit shortly thereafter.  And after reviewing my wife’s symptoms, the doctor classified her as a “yellow” case.  This diagnosis, coupled with community risk factors (my wife is a special educator for infants and toddlers,  some with underlying health issues), convinced the online doctor to refer my wife to our local PromptCare center for a test.  She was given directions to arrive 20 minutes prior to her appointment and to wait in her car until they had cleared a room for her.  She did that, and when she finally saw the doctor at the PromptCare center she was informed by the doctor that she would not get tested for COVID.  When my wife asked the doctor why not, the doctor told her that she wasn’t a severe enough case to test for COVID.  My wife challenged that statement, and reminded the doctor that she had been referred to the center not an hour earlier by a MedStar doctor who said she should be tested.  The doctor responded that the facility didn’t have enough test kits, and that they were only testing medical first-responders and individuals that were severely ill.  The doctor told my wife that her symptoms and testing history clearly indicated COVID, but the doctor was not going to test my wife because she wasn’t sick enough and not a first-responder.  She was sent home 20 minutes after being brought to an exam room.  No test.  No additional treatment.

This is absurd.  As of 7:30pm on Sunday, March 22, the state of Maryland is reporting 244 cases of COVID.  But I’m aware of three cases in my own home that will never get counted in that list, and I’m sure I’m not alone.  The government has moved too slow, been overly concerned with an individual’s direct contact with China, and has provided too few resources to accurately track the disease.  Instead of simply diagnosing the disease via the symptoms presented and mathematical modeling, doctors, at the direction of the CDC, have been obsessing over links to China while COVID spread unchecked.  It’s likely this behavior exacerbated the spread of the disease by convincing infected patients (like my son and I) that we didn’t have the disease, and that we were free to go about our lives.  We sent our son back to middle school because the doctor said he didn’t have COVID.  My wife and I took a cruise because two doctors told  me I didn’t have COVID.  And we exposed my 75-year-old parents and 68-year-old mother-in-law because our doctors told us we didn’t have COVID.

But it gets worse.  Our lack of preparedness means there aren’t enough test kits to test sick people who might have the disease; we can only test the ones that are already very ill or on the front lines.  We have no clue how many moderate, mild, or even asymptomatic cases of COVID are out in the wild.  It makes me think that we are underestimating the spread of the disease by at least an order of magnitude, and maybe more.  In other words, the response to this pandemic has been a complete failure.  I don’t trust that the government or the medical community has the faintest clue how widespread this disease is, and by extension I don’t trust the response.  The President, the CDC, the Governor, and the medical community — you have all failed me.  You have all failed my family.  You have all failed my community.  How many people died because of this incompetent response?

That last question is important for many reasons, chief among them is to motivate a much better response to the next pandemic.  So what lessons can we learn from recent failure?  The first lesson is that there are no international borders when it comes to transmission of disease.  Simply look at the routes commercial airlines fly from major hubs around the world.  From Dulles International Airport a traveler can reach most of North America and Europe, Brazil, Israel, Japan, and China via direct flights.  The same is generally true for SFO, though you can add South Korea, Australia, and New Zealand to the destinations served by direct flights.  Flights run every day of the week.  A typical 777/787 carries between 250 and 350 passengers.  For a destination like China, with major hubs in Beijing, Shanghai, and Hong Kong, that means there are typically more than 10,000 travelers per day between China and the US.  That's a lot of potential carriers of infection over the course of a month, and it only takes a couple of generations (defined as number of days / the median incubation period) before the infected have no connection to one of those travelers.  In the case of COVID specifically, this means the questions about travel to China, or contact with travelers to China, were meaningless by the end of January.

The next lesson is that data matters.  We have models for how diseases spread in a community, and these models are good predictors of how a disease will impact us, but we need a starting point.  That starting point requires aggressive testing of impacted communities so that we can understand characteristics such as communicability and incubation period.  Only then can we seed a model to understand the spread.  And once we have models, we need to run them and test them.  Geometric models tend to blow up very quickly -- consider the classic riddle about whether we should choose $1,000 a day or $1 on day 1, $2 on day 2, etc., for a month.  So when we see results from a model that blows up to tens of thousands of infected over the course of several weeks, we need to take it seriously.  We also need to validate and refine the model, and that again requires aggressive testing of the population.  Sometimes that will mean testing specifically for a disease, but at the start of an epidemic or pandemic that may be impossible.  Thus it will also require an unbiased observation of symptoms and testing to exclude other possible causes for those symptoms.

Next, preparing for a pandemic means actually preparing for a pandemic.  It may be difficult to create test kits without knowing what to test for, but we should be prepared in so many other ways.  There is no excuse for being short of masks and other protective clothing; these products can be stockpiled for years.  There's no excuse for not having a protocol for dealing with potentially infected patients before they walk into a facility.  If I walk into a clinic complaining of COVID-like symptoms, that needs to immediately trigger a response to protect the safety of practitioners, support staff and other patients in the facility.  It means having a plan in place for limiting the movement of individuals while ensuring critical infrastructure is not impacted.  A haphazard closure of schools, restaurants, and stores is not a plan.

Finally, we need some method for understanding the totality of this pandemic.  We've lost the chance to aggressively test for the infection during its active stages, and we can't just walk away from this once the worst is behind us.  At some point we will have the opportunity to test individuals for antibodies against the infection.  We need to take that opportunity seriously and, again, test aggressively.  Only by doing so can we glean an accurate understanding for how the pandemic unfolded, and we'll need that understanding to help fight the next pandemic.

For those that made it through this diatribe, thank you for your time and attention.  I wish you well, and I hope we will all be better prepared the next time.


Sincerely,


Greg

Running, Knee Pain, and Hyaluronic Acid Redux

I wrote this post back in January.  Didn't realize that it had been sitting in Draft mode ever since...

I did something this week that I haven't done in nearly a year: I ran three days without pain.  They weren't particularly good runs -- they were neither long nor fast -- but they were solid 5k's none the less, and they were pain free.

So why the change?  After a year of braces and bands, anti-inflammatories, stretching and strengthening, and finally hyaluronic acid, what made the difference?  Simple: I finally told my doctor I wasn't going to take a diuretic for my hypertension any more.

I started taking a diuretic about 15 months ago, when the other medication I was already taking wasn't adequately controlling my blood pressure any more.  The effects of adding the diuretic were nearly immediate.  In the first week I started taking the new pill, I lost over five pounds of water weight, and my blood pressure dropped by 10-11 points.  All seemed very positive.

But about three months later, I started experiencing pain in my knees.  The pain was focused at the base of my thigh, just above the kneecap.  It started in the right knee, and I just figured it was a case of tendonitis.  I didn't think much of it, so I committed to just running through it.

But over the course of about a month, the pain got progressively worse in my right knee, and I started experiencing similar issues in my left knee.  Further, the pain that had been just above the kneecap was now spreading all around the kneecap.  This was not tendonitis.

So I went to see an orthopedist about the issues I was having.  He diagnosed me with patellofemoral syndrome -- a condition where the cartilage behind the kneecap wears, and the kneecap begins to rub against the base of the femur.  The diagnosis fit, but I was surprised by the coincidence of developing the issue in both knees in the span of a month.  So I asked him if the condition could be related to the diuretic I had started taking a few months earlier.  Maybe the diuretic was drying out the cartilage, or making it somehow less thick/useful?  He gave me a look like I'd asked if my warts were caused by kissing a toad and quickly dismissed the idea.  And thus began my journey through pain meds and PT, which resulted in...not much help at all.

Not really liking the answer I got from the first orthopedist, I decided to visit another one for a second opinion.  This time I chose an orthopedist who was also an avid runner, hoping his personal experience as a runner might make him better qualified to diagnose and treat the issues I was having with my knees.  He gave me the same diagnosis (PFS) and was similarly amused by my suggestion that a diuretic might have something to do with pain in my knees.  But he at least sent me for an MRI to ensure that my kneecap really was rubbing on my femur.  It was.

While working the process with Orthopedist #2, I went back to my physician for a med check.  I brought up the idea with her as well, asking if it was possible that the diuretic could be the cause of my knee pain.  The sudden incidence of pain in both of my knees still seemed odd to me, and the only explanation that made sense to me was the new drug.  She wasn't having it.  In her mind, there was no way the pain in my knees was related to the diuretic.

So after getting nowhere with any conventional treatment, I started experimenting with hyaluronic acid pills in August.  By September I was feeling some real benefit.  I was back to running a couple of times a week, but there was still some residual pain, and I wasn't in love with the idea of taking unregulated supplements for the rest of my life.  And I wasn't shaking the notion that the diuretic was the root-cause of my problems.  In fact, the positive results I experienced from the hyaluronic acid made me more suspicious of the diuretic, since the HA pills worked my helping the cartilage attract and retain water.

When I went back to my physician last week for another blood pressure and med check, I told her that I didn't want to take the diuretic any more.  I had recently had some success reducing my blood pressure below 120/80 using a 14:10 fasting regimen, and I wanted to see what would happen if I came off the diuretic.  She agreed to let me try, though still scoffing at the idea that my knee issues could be caused by the drug.

This past Saturday morning, I stopped taking my diuretic.  By Monday I was up three pounds of water weight, and I had one of my easiest runs in recent memory.  No issues during the run, and the recovery after was better than it's been in a year.  So I ran again Wednesday.  Same thing.  By today (Friday), I was up seven pounds of water weight, and my BP is hovering at about 130/88, but I ran again -- pain-free.

The obvious question (and one I've asked myself too) is if the change could be psychosomatic?  It's surely a possibility, but there are a lot of reasons to believe it's not.  First, there were lots of opportunities over the past year where I could have willed myself better if that were the case.  Taking prescription anti-inflammatory meds didn't do it.  Neither did the PT nor the cortisone shot.  It took a good month for the HA pills to show positive results, so I'm confident this is not a psychosomatic reaction.

The other reason is that the past week has produced other obvious physical changes.  My weight has shot up seven pounds this week.  My blood pressure is up.  My face is fuller, and my fingers even felt a bit stiff last night.  There have been clearly observable changes in my body this week.  Most have been negative, but the knees are a huge positive.

And that brings me back to an off-hand comment Orthopedist #2 made back in June.  After injecting my right knee with cortisone, he worked the joint back and forth and told me, "The knee is just a large bag of fluid.  Once we get the medication in there, we need to distribute it through the fluid so it can reach the effected areas."  The knee is a large bag of...FLUID!  I suspect the fluid has been fully restored to my knees now, and they feel much better.

The moral to the story is simply don't be afraid to question what your doctor tells you.  That doesn't mean doctors are fools, and that you should discount everything they say.  But they are human, and as such they are subject to the same biases and blind spots that any of us are.  Take a scientific approach.  Take time every day to observe and understand your body.  Engage in dialog with your doctor.  Share your thoughts.  Ask questions, especially the "why" questions, and demand evidence.  Develop a hypothesis, and then work with your doctor to test it safely.  You need to know you best, and you need to be your best advocate.

Running, Knee Pain, and Hyaluronic Acid

I have been a running enthusiast for most of my adult life.  I started running in my early 20’s as a way to shed a few extra pounds one summer, but I realized once I started that running had a significant impact on my mood and clarity of thought as well.  Once I figured that out, I was hooked.  Fast-forward some 25 years, and I decided that 2019 would be the year I’d run my first half-marathon.  I’d done 10-milers before, but never had I gone further.  I felt really good in the fall of 2018, so that was my New Year’s resolution: I would spend 2019 getting in shape to run the Montgomery County Parks Half-Marathon in September.

Everything was going well until late February.  That’s when I first noticed pain in my right knee.  The pain started at the top of, or directly above, my kneecap.  I assumed it was just tendonitis and decided I just needed to do a better job warming up before and stretching after my runs.  But soon the pain spread to the outer edge of my right kneecap.  Was it iliotobial band syndrome?  I spent most of the spring experimenting with various bands and braces, trying to ease what I thought was simply a tendon or ligament issue.  But the pain kept getting worse, and it spread to my left knee as well.  It got to the point where my knees didn’t just hurt, but they burned with every step for the first mile-and-a-half of my runs.  Going up and down stairs, or even trying to stand from a sitting position, became a painful chore.  And anytime something hit either of my kneecaps, even something as innocuous as a belt buckle bouncing against them while getting dressed, sent pain shooting through my legs.

So after months of trying on my own to manage the issue with stretching, braces, and lots of Ibuprofen, I finally went to see an orthopedist.  He took X-rays of my knees and promptly diagnosed my with patellar-femoral syndrome (PFS) — basically my kneecap was rubbing against the bottom of the femur in both of my legs.  In his opinion, the issue was caused by too much tightness in my hamstrings, and what I needed was some physical therapy to stretch my hamstrings and strengthen my quadriceps.  So at the end of May, I started PT.  I spent an hour twice a week in the PT’s office, and I spent time at home several nights a week working on my PT “homework”.

Earlier in the spring, when I could no longer tolerate running three days a week, I started participating in a deep water running class at our community swim center.  The instructor was a remarkable woman in her late 70’s who participated in numerous endurance runs (24-hour type events) throughout the year.  She was also a long-time member and coach at our local Road Runners club.  Shortly after I received my diagnosis, I shared with her that I wasn’t convinced of the doctor’s opinion.  I had been doing my own research, and I was convinced that what I was experiencing was PFS caused by chondromalacia — basically a deterioration of the cartilage between the kneecap and the femur.  I really wanted a second opinion from an orthopedist who really understood running, ideally one that was a runner him/herself.  She was able to provide me some names, and I made my appointment with a highly ranked, marathon runner of an orthopedist.

On my first visit to his office, I was...not prepared.  Whatever his actual level of competence was, he clearly had a high opinion of himself.  We didn’t accomplish much during that first appointment, but he did prescribe me a regimen of meloxicam, and that did help the pain significantly.  I was more prepared for my second appointment, and we had a good discussion about my condition, courses of treatment, and outcomes.  The discussion was “good” in that we were able to communicate openly, without the overwhelming pretense of the previous appointment.  But the actual discussion of outcomes was not so good.  If it wasn’t something that would clear up after a couple months of anti-inflammatory meds, and maybe a cortisone shot or two, it was probably something I was just going to have to live with.  So yes, my self-diagnosis was correct, and the cartilage was not going to re-grow itself.  I walked out of that appointment after receiving a cortisone shot in my right knee.  I figured it was worth investigating how effective the shot would be.

The cortisone shot made a modest difference, and by the beginning of July I was starting to feel pretty good.  I was hopeful that the regimen of PT and meds was making a difference.  So I decided in the middle of July to try life without taking a meloxicam every day, and the first day without meds went pretty well.  I was a bit more achy on Day 2, but it wasn’t too bad.  Unfortunately by Day 5 I was feeling miserable again.  I stopped running.  It was back to the meds, and back to the orthopedist — now with an MRI in-hand — to discuss the next plan of action.

It was during this appointment that the doctor discussed hyaluronic acid (HA) injections with me.  The idea was that he would inject the HA formula into my knees, and that would help lubricate the joint, thereby reducing the inflammation and pain.  The doctor handed me some material on HA injections and said he would be submitting paperwork with my insurance company to get my injections approved.  He expected it would be a month or so before he had approval.

Given I had a month or so, I decided to do some research on my own.  So I started with the requisite Googling and poring over the results for research that appeared legitimate.  How effective was the treatment?  What were the side-effects and risks?  Was there anything specific to be concerned about with regards to my hypertension?  From what I could tell, the HA itself seemed rather benign.  HA is a compound found naturally in the human body.  Like most things, the body produces less of it as we age.  I couldn’t find any good long-term research (10+ years) on potential negative effects, but it seemed that the compound was generally considered safe.

What I found particularly interesting was that the research I found seemed to indicate that the results from HA injections were mixed.  There weren’t any negative outcomes, but the research didn’t point to consistent positive outcomes either.  In fact, the more I read, the more it seemed that a long-lasting regimen of HA taken orally had better outcomes than the injections.  Of particular impact was a study I found on the NIH website.   This particular study was an analysis of other studies that looked to determine the effectiveness of HA taken orally.  And it seemed to indicate that on oral HA regimen could be effective in helping knee pain.

So at the beginning of August I decided to start taking HA pills, 150mg every morning.  Why, I figured, should I let the doctor stick needles in my already-aching knees if I could get the same or better results from a pill?  I was still taking the meloxicam as well, and I decided I would continue taking the meloxicam throughout the month, as we had our family vacation planned for the middle of the month.  I didn’t feel any immediate change, and in factI felt pretty miserable after our first night on vacation.  We had set sail on the Disney Fantasy for a week-long cruise, and my youngest son spent the first evening racing all over the ship (and up and down the stairs) playing their interactive scavenger hunt.  He had me in-tow, and my knees definitely paid the price for following him.  But by the beginning of September, I started feeling better.  Much better.  So I stopped taking the meloxicam again, sure that I’d be miserable by Day 5, just like last time.  But this time, I wasn’t miserable.  Yes, I felt a little more pain being off the anti-inflammatory that when I was taking it, but it was manageable.

I had been keeping up with my PT all summer, and I decided I would try to start running again.  I did my first 5k on Labor Day morning, and I survived.  I spent a couple of weeks running 5k’s on Monday and Friday (while resuming deep water running on Wednesdays as well), and I was feeling pretty good.  So on September 20, I tried 4 miles.  I was definitely sore for a couple of days afterward, but again, it was manageable without any other meds.  And honestly, some of the soreness was probably from working muscles harder than I had worked them in a few months.  On September 27, I did 5 miles.  And to my surprise, I recovered from that run even easier than the 4 mile run the week before.  I hadn’t run 5 miles all summer!

While I was doing my research, I found out that my mother was getting “injections” in her knees as well.  Turns out she’s getting HA injections, a course of 5 over several months.  We’re comparing notes, and it seems we’re both seeing benefits right now.  But our experience raises an interesting question.  Between the two of us, we’ve visited three different orthopedists.  All three of them have suggested HA injections.  Not one has suggested oral HA.  Why?  My parents doctor told them the uninsured cost of each shot is $1500, though their insurance covers the full cost.  Meanwhile, a 30-day supply of HA capsules runs about $30.  So would you prefer $7500 a year for someone to stick a needle into your knee every few weeks, or less than $400 of oral meds?  If there’s a good reason, I haven’t heard it yet.  In the meantime, I’ll continue with the stretching and strengthening, mixing the running on land and in water, and taking an HA capsule a day.  It’s not a miracle cure by any means, but it has reduced the pain in my knees considerably.  And being able to run again has definitely improved both my physical and mental state. 

Avengers: Endgame Musings

This is not really likely to be how Endgame plays out, based on everything that’s been dropped in the media over the past 10 days or so. (No, I haven't seen, or even looked for, the leaked material. So please #Don'tSpoilTheEndgame for me or anyone else.) But with only a week or so to go before the film's release, I'll offer than maybe this is how the plot should have unfolded, based on the story presented by the 22 films leading up to Endgame.

The short answer: It was all Loki, all along.

Loki has always been among my favorite Marvel characters. The God of Mischief, for sure. But also someone who is treacherous, cunning, one to run a complex game, and certainly one to manipulate others into doing his bidding. But the MCU version of Loki, though brought to life on screen wonderfully by Tom Hiddleston, has been noticeably underpowered. Maybe that’s just a byproduct of the story telling in the MCU, or maybe that’s all been part of Loki’s long con.

Limiting the analysis to just what we’ve seen in the MCU, here’s what we know about Loki. Odin raised him as his son, and Loki clearly believed he had a shot at succeeding Odin as the ruler of Asgard and the Nine Realms. That is, of course until he finds out in Thor that he’s an adopted Frost Giant, and that Odin has been planning for Thor to succeed him for years. But Loki has ambition, and he’s not going to let his oafish brother take the throne that easy. With his brother banished to Earth in mortal human form, Loki concocts a plan to sneak the Frost Giants into Asgard to kill Odin during his Odin-sleep, betray the Frost Giants by killing Lauffey as he murders Odin, thereby starting war between Asgard and Jotunheim. Let the plotting begin.

Of course we’re also introduced to both the Tesseract and the Infinity Gauntlet in Thor. Whether truly a fake or just a retcon, the implication here is that at one point in his reign, Odin was actively pursuing the Infinity stones. Thor may not understand everything (or anything) about the stones at this point in his life, but it’s very likely Loki does. Thor is the warrior, the jock, the party boy. Loki is the kid who survived by maxing out his mind -- magic, lore, strategy, etc. We know from the opening of Captain America that Odin once possessed the Space stone in the form of the Tesseract and sent it to (left it in?) Norway a millennium ago. We also know from Thor: The Dark World, that Odin's father, Bor, once took the Reality stone (in the form of the Aether) from the Dark Elves and hid it away.  Knowing the family history with at least two of the stones, and knowing that Odin’s vault includes a relic for controlling the Infinity stones, you can bet Loki has studied them.

"Someone has been playing an intricate game, and has made pawns of us."

And that leads us to the first Avengers movie. Loki has successfully escaped justice on Asgard. He still craves the throne, but he’s also learned that Thanos already has the Mind stone in his possession, and he’s looking for more. Now Thanos would never attack Asgard, and Odin, directly without the stones. But with Odin getting on in years, and needing to enter Odin-sleep more often, a powered-up Thanos and his Black Order would be a severe threat to Loki’s plans for rule.

So how do you take an Infinity stone from Thanos? Just as one does not simply walk into Mordor, one does not simply take an Infinity stone from Thanos. But, he might be convinced to give someone the stone -- someone who, perhaps, knew the location of another stone, and offered to lead an invasion to take it and return it to Thanos. Of course, Loki had no intent to ever make Thanos more powerful. And he never had any intention of taking the stones for himself, not yet anyway. Further, if he were to take the Mind stone from Thanos -- and the Space stone from Earth -- and then not hand them back to Thanos, he’d be on the run for the rest of his days. So what better plan than to lose the Battle of New York on purpose? If that were to happen, then Thanos would lose control of both the Mind stone and the Space stone, and in theory each would probably be better protected than before.

Great plan so far. But now assume that when Loki commandeers the minds of several SHIELD agents with the sceptre at the beginning of Avengers, he realizes that SHIELD has been infiltrated by Hydra? Maybe now there’s a chance to help Hydra sow chaos on Earth (thereby keeping the Avengers occupied) by making sure that when he loses the battle, the sceptre falls into the hands of compromised SHIELD agents. Now he’s starting to clear the field ahead of him, and he’s done it by simply losing the battle on purpose!

"You faked your own death. You stole the throne, stripped Odin of his power, stranded him on Earth -- to die -- releasing the Goddess of Death."

With Odin weakening, Thanos de-stoned, and the Avengers battling Hydra and each other, it’s time to worry about Hela. He knows that when Odin goes, she’s coming back, and she was a handful for Odin even at his full strength. So he speeds Odin towards his end with magic (our first hint at how powerful Loki really is), takes the throne disguised as Odin, and basically waits for things to get so bad that Thor rushes back to Asgard. In the meantime, he’s learned that Hulk is on Sakaar and hatched a plan to use Thor and Hulk to take out Hela. So he travels to Earth with Thor just in time to see Odin pass, Hela arrive, and then he “foolishly” opens the bifrost. And how is it that Loki and Thor just happen to land on the same planet where Hulk has been hanging out, bashing heads for two years? There's really no reason to assume this was pure chance. Thor and Loki crash off the rainbow bridge at different points in both time and space, and yet both arrive on the same planet in a far-off star system. And whatever Loki's story about arriving "weeks ago", it's just as likely he's visited Sakaar before via one of the secret pathways between worlds that he found on Asgard.

Oh yeah, and Loki knows the Ragnarok prophecy too. That’s his wild card. With Hulk, Thor, Hela, and Surtur all battling on Asgard, he’s sure that one of them (and probably more) aren’t going to make it out. The most important thing for Loki is to make sure he survives the fracas with the Space stone in his possession. And sure enough, he does. So by the end of Thor: Ragnarok, Odin is dead. Hela seems to be lost in Surtur’s destruction of Asgard. Surtur’s job is done, and he’s off to whatever demons of prophecy do once they’ve completed their fated tasks. The Avengers are broken. Thanos has none of the Infinity stones. Things are looking up for the God of Mischief.

But Thanos and Thor still sit between Loki and his goal of ruling the Nine Realms (and Asgard, wherever it’s people land). So there’s still some thinning that needs to be done. Thanos isn’t going to stop trying to collect the stones, and none of the Avengers — Thor included — understand the danger like Loki does. So given the choice of backing Thanos or the Avengers, Loki puts his money on Thanos. Loki cuts a deal with Thanos to survive the snap in exchange for the Space stone -- and a little bit of theatre ensues to sell the con. He’s betting that Thanos will power up, defeat the Avengers, and then retire with his task accomplished. After all, Thanos’s goal is not intergalactic rule. It’s “saving” the universe from itself. With that task complete, Thanos doesn’t really care who rules over whom. That takes us to the end of Infinity War. Thanos is successful and looking to retire to the sunsets on a nice planet that feels like Titan once did. Half of the Avengers have been decimated. All that’s left is to take care of Thor, and then it’s on to ruling the Nine Realms.

“If we can’t protect the Earth, you can be damned-well sure we’ll avenge it.”

So that brings us to Endgame. As Tony Stark told Loki in The Avengers, “if we can’t protect the Earth, you can be damned-well sure we’ll avenge it!” And that’s where we are. In my version of Endgame, the remaining Avengers defeat a tired, wounded, unprepared Thanos; undo the snap; then then come face-to-face with Loki and…? That raises an important question. How do you create a situation where all of the now-restored Avengers and Guardians can be thrown into battle, in a way that creates drama in the outcome? Maybe Galactus, but that's challenging both from the standpoint of licensing (IIRC, Galactus was part of the Fantastic Four property licensed to Fox at the time of writing and filming) and the trap of ever-escalating baddies. So then what? Here’s a clue: we saw his head and gleaming-red eyes in Spider-Man: Homecoming. That’s right, Loki resurrects Ultron (remember, he never does the heavy lifting himself), and with Loki's help, Ultron creates a new drone army to battle Team Good at the grand finale. The Avengers and the Guardians prevail, of course. But also of course, Loki escapes in the end, left to fight another day.

That's how I'd play the story out. It provides a nice story arc for Loki, and it brings him more in line with the character we've seen in print. It also provides the "Marvel twist" on the villain. We've been lead to believe that Thanos was the bad guy from the original Avengers on -- that he was controlling and manipulating Loki, Hydra, and Ultron to advance his own agenda -- while in reality it was Loki pulling all the strings. I see it as a nice way to wrap up the character arcs of heroes and villain alike.

My very own streaming TV channel - Avengers TV!

I recently returned from a Disney Cruise, and one of the best things about it (for me anyway) was the fact that Disney has a ship-board TV channel that streams Marvel's Avengers movies 24x7. (No, I did not spend the entire cruise inside my stateroom -- I'm not THAT lame. But it was nice that for the moments when I was in my stateroom, there was awesome TV, guaranteed. And the cruise was awesome, even without the TV. That was just icing.) Even before the cruise was over, I started thinking... "How can I do this at home? It shouldn't be too hard to do." And while that was generally true, the Devil (as always) is in the details. Here's what I learned. And it should be easy enough to reproduce now, without all the busted knuckles…

DISCLAIMER:  There is potentially a murky legal issue here.  The laws in the US are a bit contradictory.  You DO have the right to make copies of copyrighted material for private, archival use (say, a backup copy if your source DVD fails).  That said, circumventing copy protection on a DVD or BluRay will put you at odds with the Digital Millennium Copyright Act.  So, what happens when a media provider denies you your legal right to make an archival copy of media you have legally purchased?  I don't know...

Hardware Requirements

Start with a small PC that supports 4K video. I tried a Raspberry Pi, but VLC doesn’t support hardware acceleration on that platform (without compiling from source, anyway), and my experience with the Pi in this case was rather poor. The streams were terribly choppy at “HD” quality.

On the other hand, I found a older NUC with a Celeron processor and a nice GPU that works like a champ. I outfitted a NUC5CPYH with 8GB RAM and a 250GB SATA3 SSD and got excellent results. Clearly not as cheap as a Pi, but in the grand scheme, it’s not too steep a price to pay. For the minimal Ubuntu load we’re going to do, 250GB is a lot of storage for video, and you might not need nearly that much. My video library is full of 2-hour+ videos at 1920x1080 with 5.1 audio, and they run between 3-4GB each. So 24 hours of video would run less than 50GB at that rate. Plus, I’m storing all my video on a Synology DS416 NAS. The SSD I used is one I actually re-used from another machine. Something like 120GB would have been plenty, so don’t feel compelled to go big, even if you’re serving the video locally.

Other things to consider are the quality of your networking gear. I run a pair of Juniper EX2300-C switches and Ubiquiti UniFi AC-Pro wireless access points in my house. I can’t speak for how well “big box store” networking gear will work. The actual video stream is not huge (maybe 1.5-2Mbps), but this setup makes use of multicast to distribute the service. So make sure you’re networking gear provides support for it — look for IGMP and IGMP snooping support -- and look for class/quality of service support as well. Anyway, something to consider…

Operating System

I built my setup using Ubuntu Server 14.04.05. You’re probably asking why. Why Ubuntu, and why such an old release? Answering in order, first, Ubuntu server is easy, or at least easy enough, to install and configure. And it’s quite light weight, especially compared against Windows, so it leaves lots of space for local video if you choose.

Now the next question: Why such an old release? The answer to that has to do with how VLC is packaged for use with Ubuntu. It seems that when Canonical packages software like VLC for an Ubuntu release, the software major release gets frozen with the OS major release. So the only version of VLC you can download for Ubuntu 14.04 via apt is VLC 2.1.x. Go to a newer (and still supported) release of Ubuntu, and you get a 2.2.x release of VLC. And it turns out that multicast streaming (or more correctly, streaming using the ts multiplexer, which is the only multiplexer supported for RTP streams) is broken in the 2.2.x releases. I know, I tested. And while the application will happily tell you it’s sending multicast packets, neither tcpdump nor external testing tools (Wireshark on a SPAN port) show any multicast data leaving the server. So go grab Ubuntu 14.04.latest from the download site. It will definitely do the job.

Additional Software

There are really just three post-install software packages we need to install on the server. The first, as we’ve seen, is VLC. In this case, since we’re running a very lean Ubuntu server (without a window manager), we’re going to install the command-line-only version of VLC. Along with VLC, we’re going to need some support for encoding and decoding audio/video, so there’s a nice package for that. Last, we’ll need a small package to announce the availability of our media stream to interested listeners on our network. Again, there’s a nice little package called minisapserver that will do the trick. Thus the complete list of add-on packages we’ll need are as follows:

• vlc-nox
• libavcodec-extra
• minisapserver

That’s it! That’s all we need on the server.

Now, what about the clients, you ask? Simple enough, we need more VLC! And the great thing about VLC is that it exists for all kinds platforms and OS’es. There are Win/Mac/Linux versions for traditional computing. There are iOS and Android versions for mobile. Heck, there’s even a version for the XBOX One now, right in the Store! So go grab the client(s) you need while you’re at it.

Procedure

Now that we’ve identified the parts, how do we put them all together?

Configure the Hardware

Let’s start with building the server. First and foremost: If your BIOS has options to enable GPU handling of video mux and transcode operations, enable them now. I can’t tell you specifically where to look in your BIOS settings, every PC seems different, but do look. Do Google. Do whatever to find the option, if it exists, in your device. And if it does exist, turn it on before installing the OS, just to be safe.

Install and Configure Ubuntu

Next, it’s time to install Ubuntu on your server. Write the ISO file to your USB stick with your favorite tool (I use Etcher), and boot the server from the USB. I won’t go through all the options during install, in most cases you can just accept the default, but I do want to point out the following:

1. When it comes to partitioning and formatting your disk, select the “Guided, use entire disk” option. Stay away from the LVM options. I’ve had lots of issues with LVM because Ubuntu doesn’t reserve enough space for the /boot volume. And it will fill up after a couple of kernel updates, if you’re not vigilant about auto-removing unused packages. So play it safe, and avoid LVM.
2. You will need to create a user account to login and manage the box. The user you create during setup will have admin control of the server via the ‘sudo’ command. The root account is locked by default in Ubuntu. So create a user account you will remember, with an appropriately strong password, for admin purposes.
3. When it comes time to select additional packages to install, you need only select the OpenSSH server. That will allow us to manage the box via SSH client, rather than needing a keyboard and monitor or the box to do so. So install OpenSSH server, and nothing else.

Once the install is complete, the server will reboot. When it does, test the login you created during the install, and bring the system up-to-date by issuing the following command from the cli:

sudo apt-get update -y && sudo apt-get dist-upgrade -y

When complete run this to clean out any unused packages:

sudo apt-get autoremove -y

By default, Ubuntu will configure your primary network device for DHCP. That’s nice and easy, for sure, but you may want something a bit more deterministic for managing the server via SSH. If you do want determinism, then you have two options. The first, and the one I would recommend, is to set up a static reservation for the server interface in your DHCP server. How to do that is an exercise left to the reader, and to Google. There are just too many variations in tools and environments to address every situation.

The second option is to configure a static IP address on your server interface. If that’s the path you choose, then your first step is to edit the file /etc/network/interfaces on the server. Use your editor of choice to make changes (nano is probably easiest for the uninitiated):

sudo nano /etc/network/interfaces

You will see a pair of lines similar to the following. Note that Ubuntu 14.04 tended to name Ethernet interfaces something like p2p1 or p128p0 -- just depends where on the PCI bus your adapter shows up. In my case, it’s p2p1:

iface p2p1 auto
iface p2p1 dhcp

Now replace that second line (iface p2p1 dhcp) with something akin to the following. (I assume you know what your IP address, netmask, and default gateway are supposed to look like):

iface p2p1 static
address 192.168.1.20
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 9.9.9.9

So that’s 5 lines to replace the ‘iface p2p1 dhcp’ line. Save your changes and close the file.

When you’ve completed all of that, go ahead and reboot the server. That will ensure all of your software updates take effect, as well as any changes you made to addressing your server.

Where is the Media?

Once the box is up and running again, it’s time to download and install the packages we’ll need to stream media. But before we do, consider where your media will be stored. Is it local -- on the server SSD? If so, then good. You can skip ahead to the next section.

If your media is resides on a network attached storage device (a NAS), then we’ll need to do a bit more server config to allow our server to mount shares from the NAS. And I’m assuming here that the NAS supports Windows-style (CIFS/SMB/etc.) shares. If that’s the case, you can follow these steps.

Create a mount point for the share on our server. I used /ds416/media on my server, so I ran the command:

sudo mkdir -p /ds416/media

In general, you can use whatever you want (so long as the directory name/structure isn’t already in use) and run the command:

sudo mkdir -p {{ path_to_my_media }}

Install the CIFS tools on the server

sudo apt-get install -y cifs-utils

Decide what credentials you want to pass to the NAS when you mount the share. (I created a read-only user on the NAS that I use for this purpose. I suggest this because our server will not need write access, and I hate open access shares. Lock it down a bit. Don’t set the bar on the ground for a potential hacker.)

Modify the contents of /etc/fstab to automount the share:

sudo nano /etc/fstab

Add the following line at the end of the file (this is all one line):

{{share_name}} {{media_dir}} cifs username={{user}},password={{password}},iocharset=utf8 0 0 

• {{share_name}} is the path to the share on the server (e.g., //myServer/media)
• {{media_dir}} is the mount point you created above (e.g., /ds416/media)
• {{user}} and {{password}} are the credentials you created on the NAS to allow read-only access from the server (You can make things a bit more secure by moving the credentials to a .smbaccess file. Look here for details: https://wiki.ubuntu.com/MountWindowsSharesPermanently

Save changes and close.

You can mount your new share now by running the command ‘sudo mount -a’. And if the server reboots for any reason, it will mount automatically.

Install the Server Packages

Now that we have media on-hand to stream, it’s time to install the streaming tools. If you remember, we’ll need the vlc-nox, libavcodec-extra, and minisapserver packages. You can install all three with one command (all one line):

sudo apt-get install -y vlc-nox, libavcodec-extra, minisapserver

Boom, done!

You can test quickly by running ‘vlc --version’ from the command line. Be sure the version reported back is something from the 2.1 release. Mine is 2.1.6.

Create a Playlist

One of the nice things about VLC is that you can feed it a playlist as a source, and it will stream the media files referenced in the playlist. VLC supports M3U playlists, which are nice because they’re just formatted text files. An M3U playlist starts with this first line:

#EXTM3U

Then it contains a bunch of media entries that follow this format. Best to start with an example and then explain the fields:

#EXTINF: 8575, Marvel - The Avengers
#EXTVLCOPT: file-caching=300
The Avengers.m4v

Now the definitions…

• 8575: The running time of the media, expressed in seconds
• Marvel - The Avengers: The author and title of the media, separated by a hyphen/dash
• The Avengers.m4v: The name of the media file on the disk/NAS. You can use a local/relative reference (as I did), or an absolute reference (such as /ds416/media/The Avengers.mv4).

Keep banging out those three lines for each media file you want to include in the playlist. When done, save the file with a .m3u extension (e.g., my-playlist.m3u) and close it out. It’s best to co-locate the playlist file with the actual media (put it in the same folder), as that will allow you to reliably use relative references for the media filenames in the playlist.

Plan the Multicast Environment

Here’s where we turn the network nerd-knobs. We need to know how to stream multicast from our server, so that: (a) we know how to receive it, and (b) we prevent potential multicast storms and/or data leakage. Things we need to set:

• Multicast group (address): We’ll pick from the range 239.0.0.0/8 for private (administratively scoped) streams. Make it something (relatively) easy to remember, like 239.239.1.1. If you ever add a second stream, make it 239.239.1.2, etc.
• Destination UDP port: It can really be anything in the range 1-65535. Traditional values are 1234 (old) and 5004 (new). Let’s use 5004 since it’s outside the range of traditionally reserved (and administratively protected) ports.
• Time to Live (TTL): Let’s set it low. This is how we control where the stream can flood. If you only have one subnet in your network (e.g., single AP, with a single network/VLAN connected to it), then you can set your TTL to 1. If you have multiple networks, separated by a router, then you’ll want to choose a TTL of 2. If you have a hierarchy of networks, separated by multiple routers, then set TTL accordingly. Just remember that TTL is decremented on every router hop, and TTL=0 means the packet won’t be forwarded.
• Type of Service (TOS/DSCP, this is optional): If you want to mark your streaming traffic as real-time traffic, then we’ll use the hexadecimal value 0xC0 (that’s zero-hex-charlie-zero).

So let’s test. Let’s stream to rtp://239.239.1.1:5004/ with a TTL=1 and DSCP marking of 0xC0. On the server, type the following command (all one line):

cvlc -v {{path_to_media/playlist.m3u}} --sout '#rtp{mux=ts,dst=239.239.1.1,port=5004,ttl=1}' --sout-keep --loop --dscp 0xC0

That should spit a whole bunch of log messages at you, but it should come to a rest after a few seconds. If it continues to stream log messages at you for 15 seconds or more, then hit CTRL-C and scroll back through the messages to see what the issue is. Typically at this point it’s a syntax error on the command line -- wrong filename/path, something misspelled, missing double-dash, etc. Review carefully.

Once things to quiet down nicely, grab your favorite client. Open VLC, select Network Stream, and enter the URL rtp://239.239.1.1:5004/ when prompted. In a few seconds, you should see your video streaming on the client. If not, time to review log messages on the server again. Rinse and repeat until you can see your video stream on the client.

Start the Stream as a Service

Now that you've had a successful test from the command line, it's time to make it all happen automatically.  Start by creating a user account that will simply be responsible for the operation of VLC:

sudo adduser --system --home /etc/vlc vlc

That will create a system account named vlc, with a home directory of /etc/vlc.  Now lets put something in the vlc user's home directory, namely a script to automatically launch the stream.  Using your favorite editor, create a new file named /etc/vlc/start-vlc.sh and add the following lines to the file:

#!/bin/sh
sudo -u vlc {{command line you ran in the test, and worked}} > /dev/null

Save and close that file.  Now set the ownership and permissions correctly:

sudo chown vlc /etc/vlc/start-vlc.sh
sudo chmod 755 /etc/vlc/start-vlc.sh

Next, open up /etc/rc.local and add the following lines, just before the exit 0 at the end:

# Start streaming with vlc
/etc/vlc/start-vlc.sh &

Save your changes and exit.  The stream should now start automatically if/when you reboot the server.  Go ahead and try it out -- reboot the server.  Once it's back up and running, run the command:

ps ax | grep vlc | grep -v grep

That should return about four lines/processes in the output, two of which will contain a string that looks very similar to the command string you tested with.  If that's the case, fire up your VLC client and verify that you can see the stream.  If you can't, or if the output from your ps doesn't look right, then go back and double-check the /etc/vlc/start-vlc.sh and /etc/rc.local files.  If there's no obvious syntax error, you can run the start script yourself to see what happens:

sudo /etc/vlc/start-vlc.sh

Of course, if that works, your problem is probably in the /etc/rc.local file.  If it doesn't, look at /etc/vlc/start-vlc.sh.  You know the routine: tweak, test, repeat until it works...

Configure the SAP Server

Once you've successfully started the stream automatically, it's time to advertise your stream to interested clients.  The SAP server is responsible for that.  We've already installed minisapserver, so all that's left is to configure it.  So open /etc/sap.cfg in your favorite editor, and set the following:

sap_delay=5
interface={{ifname}}

Replace {{ifname}} with the name of your streaming interface.  Again, in Ubuntu 14.04, it probably looks like p2p1, or something similar.  Under the [program] section, create an entry similar to the following for each stream you want to advertise:

[program]
name={{ friendly name for channel/stream }}
user=videolan
machine={{ server IP address }}
site=-
address={{ multicast address you used in the cvlc command }}
port={{ port number you used in the cvlc command }}

Again, you will need one [program] entry for each stream or channel you want to advertise.

Next, edit the /etc/default/minisapserver file and make sure RUN=yes is set in the config file.  Save and close, if necessary.  All that's left is to (re)start the SAP server:

sudo /etc/init.d/minisapserver restart

That's it!  Crack open a cold one, open your VLC client, and browse the Local Network option.  You should see your stream advertised there.  Click to watch!

Final Thoughts

This is not a perfect process.  And in fact, lots of things will be broken in different ways.  What I've found so far...

First VLC really is awesome.  It's a powerful tool that can be used to both transmit and watch media files, from lots of sources, and on lots of platforms.  But for all its awesomeness, it is a volunteer-based, open source project.  And as such, it has some warts:

• We've already discussed that VLC for Linux, versions later than 2.1.x, has issues streaming with the ts-mux.  If you try a 2.2.x release of VLC for Linux, it will break streaming!
• Multicast in VLC for Mac is completely broken.  I'm testing with version 3.0.0.  It will not stream.  And when I try to receive I multicast stream, I can confirm that VLC never sends an IGMP join for the multicast group (address) that contains the video stream.
• The XBOX One app is a pretty nice client, but you can not actually click-to-start a stream that shows up as a SAP advertisement when you browse the Local Network.  You can manually add the stream by typing in the URL in the Network Stream source.
• The Android app behaves a lot like the XBOX One app.  You can manually add the stream, but you can't tap-to-launch from the SAP advertisement.
• The iOS app seems to be the nicest client.  You can actually tap the SAP advertisement under Local Network, and it works as expected.  So start here, if you can, for maximum enjoyment.
• I can't speak to the Windows client (sorry).

Streaming multicast over WiFi can be a bit challenging as well.  I saw really poor performance with my Ubiquiti UniFi system when I started.  I did some Googling that seemed to indicate that you needed to set Multicast Enhancement in your WiFi advanced options.  I'm really not sure why that made the performance better (it seems to only enable IGMP snooping), but it does.  There's this odd statement on the help for the Block LAN to WAN Multicast and Broadcast Data option that says, "Multicast/Broadcast data is sent out at the lowest modulation rate..."  So, maybe by turning on IGMP snooping, you're telling the AP to use better modulation rates if there are interested listeners?  Just a guess, I don't know for sure.  But if your WiFi solution provides options for improving multicast performance, I suggest enabling them.

IGMP snooping has some issues as well.  Basically, switches that perform IGMP snooping seem to assume that the multicast stream does not originate on the local subnet, or if it does, it originates on a switch connected to the IGMP querier.  Because of the way my network is connected between the ISP NID (my router is in the basement, close to the NID) and my office (where all the servers and storage are, three switches away), this is not my setup.  Unfortunately, IGMP snooping will absorb a join request from an interested host and build the multicast tree towards the router (IGMP querier) even if the source is in the other direction.  So you can turn off snooping, in which case the multicast stream is flooded to all switch ports on a VLAN with interested listeners, or you can create static joins in the downstream direction.  As I don't have many streams to manage (at least not now), I chose the latter.

Well, that's it!  It was a lot of fun putting the system together.  I got the chance to exercise some sysadmin skills, some networking skills, and in the end I have an private, in-home TV channel that streams Avengers movies 24 hours a day, seven days a week.  What better reward could there be?

My Quest to Demonstrate EVPN Multihoming to the Server in a Virtualized Data Center Topology

I recently had a customer ask if I could help him develop a proof of concept to illustrate how EVPN multihoming could be used in a datacenter environment to replace technologies like Link Aggregation Groups (LAG) and Multi-Chassis LAG (MC-LAG) to support connections to dual-attached servers.  Eager to prove the concept out, and lacking in physical hardware to rapidly build and test the topology, I decided to try to implement the POC in a KVM environment, with Wistar acting as the topology manager for the environment.  Fundamentally, I was building what has become a "standard" layer 3 Clos datacenter fabric composed of Juniper virtual QFX switches attached to some Ubuntu servers.  Seemed simple enough, but clearly I hadn't thought through all the details...

Standing up the virtualized physical topology was simple enough -- two vQFX spine switches, three vQFX leaf switches.  Connect one server to Leaf #3 by a single Ethernet interface.  Connect the other server to Leaf #1 and #2 -- a single Ethernet interface from the server to each switch.  (Just for fun, I planted a Juniper virtual MX at the top as DC edge router.  Not necessary for the POC, but still fun to play with.)  In all, the topology looks like this:

I had previously written an Ansible playbook that automagically built the configs for each of the Spine and Leaf devices in the topology.  It wasn't perfect, as it only accounted for single (rather than dual) attached servers, but it got me to 98% configured in five minutes.  All that was left was to configure the aggregated Ethernet (ae) interface and its respective child links on Leaf #1 and #2.  Simple.

I started the topology and quickly realized I had a problem.  I was running standard Juniper LAG on the vQFX leaf nodes -- basically that means setting LACP active on the AE interface.  And I had set up bonding on Server #1 for mode 4 (LACP) with hashing set to layer3+4, as close to an analogous config as I could get on the server with respect to the Junos configuration.  But the AE link was down on the switch side...  Careful inspection revealed that no LACP frames were reaching the server.  The vQFX switches were sending them, but the server was not receiving them.  Nor was the server actually sending any LACP of its own.

It took a few seconds for the input to process...  The server was not connected to the vQFX leaf nodes via virtual wires, it was connected via Linux bridges!  And LACP frames are of a format (01:80:c2:...) that 802.1D compliant switches do not forward.  So the LACP exchange between the leaf nodes and the server were being swallowed by a couple of Linux bridges...

The good news is that there's a fix for that!  Since the 3.2 kernel, developers have included a little tool called a Group Forward Mask that allows you to direct the bridge to ignore (and hence forward) certain layer 2 protocols.  You can write to the Mask in this way:

echo maskValue > /sys/class/net/brXXX/bridge/group_fwd_mask

where maskValue sets the bits in the lower half of the MAC address that you want the bridge to ignore/forward, and brXXX is the name of the bridge where you want to implement this change.  Simple, and the scope is reasonably limited.  So I just wrote 255 to the mask so that my bridges-in-question would just ignore any values in the last octet and forward all of the potential "interesting" traffic (Spanning Tree, LACP, LLDP, etc.) onwards.  Except it didn't work.  I tested with LLDP, and that worked beautifully.  But I still couldn't get my Linux bridges to forward LACP frames.  Hmm...

So a little more reading turned up something else.  The folks who implemented the group_fwd_mask change were afraid of folks like me, and they were concerned that allowing too many protocols (like Spanning Tree and LACP) through could be disastrous -- and they're absolutely correct.  So they implemented another feature in conjunction with the group_fwd_mask -- a #define named BR_GROUPFWD_RESTRICTED that is set to 0x7u, which prevents you from modifying the lowest three bits in the forwarding mask.  So you can't change the bridge behavior to permit Spanning Tree (01:80:c2:00:00:01) or LACP (01:80:c2:00:00:02), among others.

Now it's on.  That define is contained in the Linux source tree in the net/bridge/br_private.h header file.  So, add the Linux source package to the hypervisor platform, recompile with BR_GROUPFWD_RESTRICTED set to 0x0u, and reboot on the new kernel.  Then "echo 255 > /sys/class/net/t5_br9/bridge/group_fwd_mask ; echo 255 > /sys/class/net/t5_br10/bridge/group_fwd_mask" and go.  Take that people!

That only half-fixed the issue.  At this point, I could see LACP frames from the vQFX leaf nodes reaching Server #1.  Literally, looking at the output from tcpdump -e -i ens4 (one of my child links on the server) showed perfectly-formatted LACP from the vQFX reaching the server child link.  But the output from "cat /proc/net/bonding/bond0" seemed to indicate that the server wasn't actually processing the frames.  The syslog output seemed to corroborate that as well, as no child links were joining the bond.  And on top of that, the server was not sending any LACP frames.  And yet, LLDP was working great.  And if I forced the vQFX side up by committing "set interfaces ae0 aggregated-ether-options lacp force-up" then I could pass traffic between Server #1 and Server #2.  Weird...

I spent a couple more days occasionally Googling variations of "lacp" and "ubuntu" with "active mode" and "not receiving" to look for answers.  And I tried all permutations of configuration in /etc/network/interfaces where I did or did not add explicit slave devices to the bond interface, and where I did or did not assign a bond master to the child links ... ifdown/ifup combinations ... reboots ...etc.  Then I found this statement that was both odd and obvious on The Geek Stuff Blog:

If the Speed, duplex & Link status is unknown then the interface may be in down status. Try to bring up the interface using “ifconfig up”. If you still do not see the link then the interface is not connected to the switch.

I knew my server and switches were properly connected through the Linux bridges, and they were clearly working, as it seemed I could pass everything BUT stinkin' LACP over the links.  But what did I have to lose?  I checked the output from "ethtool ens4", and sure enough, ethtool reported a whole lot of nothing:

Settings for ens4:
Supported ports: [ ]
Supported link modes:   Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes:  Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Link detected: yes

So back to check the config for my server Ethernet ports.  It seems that when Wistar built the topology, the network interfaces were created as type virtio.  That clearly wasn't working, so what about good ol' e1000?  Shut the server down, changed the AE child links to type e1000 and rebooted.  Now the ethtool output looks like this:

Settings for ens4:
Supported ports: [ TP ]
Supported link modes:   10baseT/Half 10baseT/Full
                       100baseT/Half 100baseT/Full
                       1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes:  10baseT/Half 10baseT/Full
                       100baseT/Half 100baseT/Full
                       1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x00000007 (7)
      drv probe link
Link detected: yes

Well that's a lot better!  And all of the sudden, the output from "cat /proc/net/bonding/bond0" shows healthy data.  And my vQFX's see LACP from the server.  Rolling back the vQFX configs to drop the "force-up" change, it all still works.  Flawless!  I've been running Ping, SSH, SCP, etc. without issue ever since.  And I can even see the various sessions being load-balanced across the child links in the bonded interface!

So there you have it.  While literally nothing else cares about the speed and duplex settings reported out by ethtool, LACP cares.  And without real data to indicate that the links are up, Linux will not do any LACP processing.  And if you want to build your own EVPN multi-homing POC on KVM, remember these important points:

1. You will need to build your own kernel to tweak the BR_GROUPFWD_RESTRICTED define so that you can manipulate the lower three bits in the group_fwd_mask.
2. Running that kernel, you will need to write a value to the group_fwd_mask for the correct Linux bridge that directs it to forward LACP.  Do this with great care, as there is a reason why 802.1D bridges do not forward this traffic by default.  Best to ensure that the switch in question only has two connected devices/interfaces -- the ones at each end of your virtual wire.
3. You will also need to be sure that the virtual network interface you use on your virtual Linux hosts properly reports interface state in ethtool.  In my case, virtio did not, but e1000 did.

Happy hacking!