Modeling Data With Functional Programming In R – Chapter 1


, ,

I’m pleased to announce that a rewrite of my first chapter of “Modeling Data With Functional Programming In R” is available in draft format. Most of the feedback of the previous version centered around accessibility and practicality of the material. In response, this first chapter is dedicated to discussing the functional programming paradigm and the key role of the function. It then progresses into talking about the relationships between mathematical notation we take for granted and functional programming syntax. The chapter concludes by introducing the lambda calculus using Church numerals to illustrate key concepts. This version includes many examples using the iris dataset, so the examples are all easily reproducible.

Any and all feedback is greatly appreciated.

Rowe. Modeling Data With Functional Programming In R, Chapter 1

Opening the time capsule: the Mimetic Sequence series of sculptures


, , , ,

This is the third “time capsule” post, highlighting some of the artwork I did around the turn of the millennium. See the first post on my Bitstream series and the second post on my Identity series.

While I was living in Bushwick a friend was living in a loft space that used to be a mirror factory. When the factory owners left, they also left boxes full of mirrors. This seemed to be routine when people vacated a building, as the loft space I was in used to be a doll factory, and the basement was filled with doll molds. One defintion of creepy is rummaging through a dark basement and encountering thousands of rusted molds of body parts. As molds, the heads in particular were creepy since they only had holes for the eyes. Anyway, my friend knew I experimented with different materials and asked if I wanted any mirrors. Of course I said yes, and I drove up, loading as many boxes as I could without destroying the suspension.

Armed with all sorts of mirrors, the question became what to do with them? I tend to look at things differently from others, and the mirrors were no exception. Rather than using them as is, I decided that mirrors have much more interesting textural and reflective properties when broken. Around this time I was also playing around with concrete. A few years later, I developed this series, which is a culmination of my work with broken mirrors and working with molds and casts.

Artist Statement

In man’s conquering nature, technology is used as the medium of choice. Out of ever increasingly complex devices and structures, man recreates nature, yet this new version is strikingly dissimilar to the effortless beauty that nature produces. In modeling nature, we strip it of its organic essence, imparting upon it a stark humanism.

moss i + ii

Date: 2002
Materials: Polyester resin, mirrors, dirt
Dimensions: 297 mm x 297 mm x 18 mm, 297 mm x 297 mm x 19 mm



sustenance i + ii + iii

Date: 2002
Materials: Polyester resin, pork ribs, mirrors
Dimensions: 190 mm x 100 mm x 60 mm, 190 mm x 140 mm x 60 mm, 190 mm x 140 mm x 65 mm




Date: 2002
Materials: Aluminum, plexiglass, bolts
Dimensions: 2′ x 2′ x 2′



Opening the time capsule: the Identity series of paintings


, , , ,

This is the second “time capsule” post, highlighting some of the artwork I did around the turn of the millennium. See the first post on my Bitstream series.

For those following along, of the materials I had used in the Bitstream series, I was most captivated by plexiglass. At work I became known as the plexiglass guy because I would walk in with found plexiglass a lot of the time. I discovered that it’s good to be known for obsessions, because it’s easy for people to be nice to you since they know exactly what you like. Case in point, a friend of a friend was moving into a new office, and the previous tenants had left a 4′ x 8′ x 1″ slab of plexiglass behind. Since she knew about “some guy” that loves plexiglass, I was given first dibs to pick it up. Of course I said “yes” and recruited a colleague to help me bring it to the office. Well, it turns out that a 1″ thick sheet of plexiglass is really heavy. So here’s me and this other guy walking up 5th Avenue near the Flatiron with this 4×8 foot sheet of plexiglass. I’ve done a lot of things to get weird looks, but this was the first time the weird look was followed by very fast dodging! We were like a semi-truck lurching down the sidewalk, and I think people knew based on the strained look on our faces that we weren’t stopping for anyone.

Needless to say we managed to bring that monster into our office without harming the plexiglass (or any people), and the boss was kind enough to let me keep it there until I could figure out how to transport it to my apartment. I eventually realized that I had no choice but to cut it up. This was troubling for me, because how could I possibly know at that time how I was going to use it? Well, I had to roll the dice and do the best I could. I sketched out some rough ideas, worked out dimensions, and came in one weekend with a circular saw. Most of the plexiglass I had worked with up until then was 1/4″, which is easy to cut. It gets a little hot, but it’s not too big of a deal. One inch plexi is a different story altogether. My poor circular saw had a heck of a time as the blade would heat up, melt the plexi, get thrown about, and then cool on the blade. This process had the unfortunate effect of jamming up the saw pretty badly. So then I had to stop cutting, wait for the blade to cool off, use a screwdriver to pry off the stuck plexiglass on the blade (all the way around), and start over. Not only was it tedious, but the screech of the blade and the fumes generated by the melting/smoking plexiglass made for quite a hostile environment. Thankfully, when I’m hyper-focused tedium disappears while my tolerance for horrible conditions increases dramatically. It’s remarkable that I didn’t set off any fire alarms or piss off the neighbors. I do remember people coming to work on Monday wrinkling their nose and asking what that awful smell was, not to mention where all those while plastic flecks came from. I tried to be nonchalant about it, but I’m pretty sure they all knew who the culprit was.

Artist Statement

What is the relationship between the mathematical concept of the identity and the language concept of identity? In math, I is the identity if for any A, I * A = A and A * I = A, where * is some operator. This series attempts to juxtapose the meanings of the two identities, whereby the distinction must be made when viewing the piece: which participant is the identity?

Embedded within this discussion of the identity is the classic dialectic between Parmenides and Heraclitus; one must ask what is the significance of change and not change when interacting with the piece?

identity 7D0.2 (tribute to eva hesse)

Date: 2000
Materials: Aluminum, steel cable
Dimensions: 2′ x 2′ x 8′



identity 7D2.2

Date: 2002
Materials: aluminum, plexiglass, steel cable, bolts
Dimensions: 3′ x 3′ x 5′



identity 7D2.3

Date: 2002
Materials: plexiglass
Dimensions: 1344 mm x 742 mm x 24 mm



identity 7D2.B (in perpetuity)

Date: 2002
Materials: plexiglass, mirrors, steel cable
Dimensions: 1340 mm x 742 mm x 43 mm


Opening the time capsule: the Bitstream series of paintings


, , , ,

This is the first post of a three part series highlighting artwork I made at the turn of the millennium. See part two here and part three here.

Most people don’t know that after graduating from college with a double major in engineering and math that I moved to New York to become a painter. It was around this time that I started transitioning from image-driven paintings to things totally abstract. An instructor I had in college both inspired me and also fueled a bit of incredulity as he was known for devoting close to a decade painting only birch trees. He was fascinated by the texture of the bark and how to emulate that in oil. It probably took me a good 8-10 years to fully appreciate his direction, and when I did I went off the deep end.

This series of studies combines my personal exploration of texture melded with my thoughts about digital information. As much as I enjoyed painting, I actually found the traditional process a bit tedious. I was more energized by throwing paint around, creating my own brushes out of weird material, painting using water (but not watercolors), cutting stuff up, smashing stuff, setting things on fire, etc. Eventually I started thinking about how to create texture without paint, which is where this series leads.

From the conceptual side I wanted to imagine what it meant to anthropomorphize data. So much talk of cyber-this and cyber-that obviously leads to questions like those explored in numerous science fiction novels and movies. Things like Blade Runner, where we anthropomorphize and then fall in love with robots, or Total Recall, where we don’t know which memories are real. If you peel back the layers, eventually you get to the question of what about the data itself that powers the virtual realities? Why not anthropomorphize the actual 1s and 0s? These pieces explore this idea using various materials to represent binary information.

Naivete of a bit stream

Date: 1999
Materials: Wood, plexiglass, sheet metal, latex paint
Dimensions: ~26″ x ~11″ x ~6″



Malfeasance of inverted personality

Date: 1999
Materials: Wood, plexiglass, sheet metal
Dimensions: 37.5″ x 27.5″ x 3.5″


Misplaced title 1

Date: ~2000
Materials: Wood, plexiglass, sheet metal, caulk
Dimensions: N/A


Misplaced title 2

Date: ~2000
Materials: Wood, plexiglass, shower curtain, latex paint
Dimensions: ~28″ x ~11″ x 3″



Misplaced title 3

Date: ~2000
Materials: Wood, fiberglass, sheet metal
Dimensions: 24″ x 24″ x ~16″


Central Park at night in wide angle


, , ,

I recently bought an ultra wide angle lens for my camera, so I’ve been looking for opportunities to take pictures. My first foray was in the woods at the site of a collapsed house. The odd angles worked great with the wrapping distortion of the fish eye. This time around I thought I’d explore the cityscape. What better place than Central Park.

First is the area around Columbus Circle. I know fisheyes can be cheesy, but sometimes it can be used to enhance a composition.



Inside the park, the daffodils were out en masse. They are probably my favorite flower as they seem so cheerful and come in so many varieties. Regardless of my mood, a daffodil will always put a smile on my face (or at least on the inside). No other flower has this effect on me. This picture is looking south with the Essex House in the background.


This is the Gazebo near Cherry Hill at The Lake looking west. It’s quite a dramatic change from the still lively crowd around Columbus Circle to the almost pure tranquility in the park. I guess for most people there isn’t much to do, but I took my time soaking in the quiet. I even managed to surprise some ducks and geese who also weren’t expecting anyone out and about.


Here’s the north side of The Lake again looking south. On the way here I surprised a raccoon. It’s surprising how much wildlife is actually on Manhattan. Up in my neck of the woods (northern Manhattan) we’ve even seen a wild turkey.


So next time you’re out late, take a stroll through Central Park and marvel at the nocturnal scenery. Just remember to be quiet and don’t scare the wildlife!

The allure of the Highland cattle


, ,

On my way to a hike near Albany, I drove past a field with some Scottish Highland cattle. At first I thought they might be yaks, but it turns out that they are a breed of cattle known for their long, wavy coats. The so-called red Highlands have particularly nice coats. Judging from the hair on its head, I’m guessing that this is where the moppy-style haircut originates from.


This one seemed curious and approached me as I was taking pictures. I didn’t realize that it was sticking its tongue out, so perhaps it was mocking me instead!


The females apparently have horns as well. Here’s a tender moment with mother and calf. I felt a little bad taking these shots as the calf seemed bothered that I was pointing a long lens at it during feeding time. It would pause, look at me, turn, and look at Mom. I didn’t think babies were that self-conscious about these sorts of things, but maybe they are a sensitive breed.

Here they are both looking at me. (Unfortunately the picture is a little blurry. Oh, to have auto-focus.)


Eventually the milk ran out or it got tired of the paparazzo, and it wandered over to its friends. I was surprised by how much thicker the coat of the dun Highland was.


Here’s Mom after feeding time.


I eventually went back to my car to retrieve a different lens. Upon my return, a number of the cattle started approaching me, which I thought was great fortune. It turns out that they really did tire of my presence as they started braying loudly. It was a rather awful sound and nothing like a ‘moo’. The only thing separating the half dozen head and me was a thin, electrified (I’m guessing) wire. So rather than push my luck, I chose to leave quietly, which probably means that I would make for a poor paparazzo.

How baby names propagate as memes across the US


, ,

A good friend of mine sent me a link to an animated gif that shows the popularity of baby names across the US over time [1]. The visualization uses the Social Security Administration records to display the most popular female name in each state from the early 1950s through present. It’s interesting to see how the top names spread across states and eventually decline in popularity. Unfortunately my name isn’t the correct gender nor popular enough to make an appearance. Fueled by equal parts ego and curiosity, I wanted to see how a single name spreads across the nation based on it’s prevalence in each state. For instance, a common name like Brian never tops the list but still has ebbs and flows of popularity that are interesting to track.

An Interactive Visualization

Hence I created a visualization that does just this. It uses the census data [2] from the early 1900s up until present, so any name present in that dataset can be visualized. There are more than 29,000 names available, not to mention some appearing in both genders. Instead of ranking the occurrences, I use the prevalence of a name (for a given gender and state) to determine the levels of the choropleth. The prevalence is thus binned into eight quantiles, plus a ninth for no data (i.e. 0 occurrences). This way you get a sense of the change in popularity of the name and how it spreads from state to state. This method provides deeper insights and more opportunities for exploration than simply viewing the top ranking names.

Geographic clustering

In fact, what is most interesting is to see how some names are popular in a specific geographic region while others are more broadly popular. Presumably the way names spread are governed by demographics, technology, and media. This is illustrated in the name Terrence, which is first popular in the Midwest in the 40s and 50s.

terrence - 1946

By the 80s, the name has declined in the Midwest and has become more popular in the South.

terrence - 1987

Media Influence

Other names are clearly influenced by media and popular culture. Go ahead and view the prevalence of Trinity (female) and Neo (male), both of which come onto the scene after the first Matrix movie. Similarly, the name River (male) becomes popular after the death of River Phoenix.

Gender Differences

Another insight is the difference in prevalence between the genders for a given name. Avery as a male name mostly appears in the South. By the 1950s it has spread North but not West. It isn’t until the late 1990s that the name has spread across most of the country.

Interestingly enough, the region that took the longest to be adopted is the same area where Avery is predominantly a female name. As a female name, it isn’t until the mid 1980s that the name becomes broadly considered a female name.

Other thoughts

I’m a sucker for inverse processes, so a natural follow up is to ask whether it is possible to predict age and state of birth based on a name. Some of the more unique names are so rare that you can deduce a person’s age and state of birth rather easily. Another question worth pursuing is what effect television had on the choice of names.

So what does all this mean for Brian? Sadly, Brian’s golden age was in the 70s and has been in decline ever since.

Implementation Notes

There’s nothing too fancy in this visualization. The visualization itself is implemented in d3.js, while the data was processed in R. The source data from the SSA is CSV format. I use my Odessa package to convert this to a JSON-compatible structure. The reason is that behind the scenes I use the RJSONIO package, which unfortunately marshals data.frames in column-major format. Here’s an example:

> toJSON(data.frame(x=c('a','b','c'), y=1:3))
[1] "{\n \"x\": [ \"a\", \"b\", \"c\" ],\n\"y\": [ 1, 2, 3 ] \n}"

Most JSON representations of table structures are row-major. In this format, an array represents the table where each element of the array is a JSON object. Each object represents a record/row in the table. Hence, in Odessa, I provide a function called row_major that does this conversion.

> toJSON(row_major(data.frame(x=c('a','b','c'), y=1:3)))
[1] "[\n {\n \"x\": \"a\",\n\"y\": \"1\" \n},\n{\n \"x\": \"b\",\n\"y\": \"2\" \n},\n{\n \"x\": \"c\",\n\"y\": \"3\" \n} \n]"

Odessa has a number of useful functions for data management. Coupled with the recently published, they provide a powerful paradigm for working with data.


[1] Map: Six Decades of the Most Popular Names for Girls, State-by-State
[2] Social Security Data

How to estimate the total cost of healthcare and why metal levels are useless


, , ,

This is part 2/3 in my series on healthcare costs. The analysis is driven by my own needs as an entrepreneur to find affordable healthcare. Part 1 explains why health exchanges are not well-functioning markets, part 2 addresses the issue of price transparency, and part 3 introduces a web app that estimates your annual healthcare costs.

The metal levels of Bronze, Silver, Gold, and Platinum (plus a fifth non-metal level, Catastrophic) were (presumably) introduced to help consumers compare different healthcare plans. The basic idea is that the metal level determines the balance between premium, copayment, and total out of pocket cost [1]. Behind the scenes, the metal level specifies the actuarial value of the plan, which is useful for insurance companies, but less so for buyers of insurance. The conventional wisdom is that if you go to the doctor a lot, then you should get a Gold or Platinum plan that has lower out-of-pocket costs, while you should get a Bronze or Silver plan if you don’t. This theory works well when employers paid most of the premium, but in the model where an individual pays the whole premium, the conventional wisdom doesn’t hold. Even worse, the metal levels tell you nothing about the expected annual cost of healthcare. Since metal levels also are not an indication of quality of service, they have no purpose other than to confuse people.

In the rest of this post, I detail a brief study I conducted on a subset of the health insurance plans available in New York via New York’s Official Health Plan Marketplace. I also provide some advice on estimating your annual healthcare costs, which lays the groundwork for a web app I’ll introduce in part 3. Note that none of the examples are endorsements for a given plan and are provided for educational purposes only.

Bounding Healthcare Costs

What exactly is the total cost of healthcare? I define this as being the net sum of premiums, doctor’s visits, medication, etc., up to the out-of-pocket limit. Note that none of the 66 health plans available on the exchange provide out-of-network coverage, so the below analysis assumes exclusively in-network care. This analysis is also based on a family with no children. For single individuals the cost will be lower, while for families the cost will be higher.

While each family’s total cost of healthcare will vary, there are some absolute lower and upper bounds to the cost that apply to everyone. The minimum anybody pays for healthcare will be their annualized premium (i.e. 12 * monthly premium). On the upper end, the maximum anybody will pay will be the annualized premium plus the out-of-pocket limit. Let’s see what this looks like across the various health plans (click to enlarge).

Annual cost of healthcare

This chart shows the annual premium for each plan plus the total out-of-pocket costs. The plans are then sorted by max total cost. On the face of it, the fact that a Platinum plan has the lowest total cost supports the conventional wisdom. Yet on the other end, the two most expensive plans are Gold! We also find that there are Bronze plans that have upper bounds that are less than Gold plans, despite the higher out-of-pocket limit.

The reason this is the case is that premiums matter. A lot. On average, the premium accounts for 46% of the total cost. The actual amount varies by plan, where Platinum plans are in the 70% range while Bronze are in the upper 30s. Couple this with the fact that the upper limit to the out-of-pocket limit is mandated by Federal law [2], you can imagine that the lower the premium, then the lower the overall cost.

And yet the 9 cheapest (in maximum annual cost) are Gold and Platinum plans, which have significantly higher premiums. How can this be? The reason has to do with probability. Keep in mind that the probability of paying at least the annual minimum is 1, but the probability of paying the annual maximum is significantly less than 1. So the likelihood of you paying the annual maximum is relatively low, unless you suffer from major illness or disease.

Modeling The Cost of Care

Now that we know the upper limit isn’t a good indicator of our expected annual healthcare cost, we need to look for something better. There are numerous ways to model this ranging in level of sophistication. I chose a simple analysis based on how I assumed others would think about healthcare: about how many times do I see the doctor and specialists? I’m assuming that for most people these are the primary contributors to healthcare costs. (A more thorough analysis would introduce additional factors.)

In this model it is easy to construct a cost surface based on the number of doctor visits and the number of specialist visits. The absolute minimum cost is again the annual premium. This is what it looks like for EmblemHealth’s Platinum plan, which has the lowest annual maximum cost.


To understand this graph, the surface represents the accrued cost of healthcare based on the number of doctor and specialist visits. If you don’t go to the doctor at all, then the cost is the minimum point on the graph, which is equal to the annualized premium. Notice how the cost flattens out once the out-of-pocket limit is reached. I discovered that it’s really only the Platinum plans that reach this limit. Obviously if you factor in hospital stays and non-trivial surgery, these limits would be hit very quickly. But then you wouldn’t need this analysis as you would just look at the annual maximum.

Now look at EmblemHealth’s Gold plan. With the same number of doctor visits, the out-of-pocket limit is not reached, yet the total cost is essentially the same. However, the premium for the Gold plan is $921.70 versus $1108.44 per month for the Platinum. Hence, with the Gold plan, you’ll likely save about $2000 per year unless something catastrophic happens. In this case, you’ll spend about $2000 more than if you had the Platinum plan.


Each provider and plan has a different surface, which is dependent on their cost structure, particularly whether there are co-pays in addition to coinsurance. Another factor is due to the number of possible paths of acquiring service. As the number of doctor and specialist visits increases, there are more possible scenarios. Consider a situation where you see the doctor once and s/he tells you to go see a specialist. You see that specialist 4 times and then follow up with your primary care physician. Now suppose you were already seeing a specialist, so you see the specialist 4 times, followed by a doctor’s visit. Then you have your annual physical. While each of these scenarios results in 4 specialist visits and 2 doctor visits, the amount applied to your deductible is different. Hence the boundary that defines when the deductible is met will be different for each run of the simulation. As an example, this is the Empire Blue Cross/Blue Shield Silver plan.


For comparison, here is one of Empire’s Gold plans, which has a flat rate for all primary care doctor visits, which doesn’t contribute to the deductible. The effect of this is that the surface is now deterministic (and smooth), meaning that the deductible is completely determined by the number of specialist visits. I return to this in the Methodology section below.


The lesson here is that based on strictly primary care and specialist visits, it is unlikely that the out-of-pocket limit will be met unless something catastrophic happens. For one final comparison, here is what a Bronze plan looks like. This is for Health Republic, a new non-profit co-op provider.


Again we see a tapering effect once the deductible is met. However the change is not as pronounced since the coinsurance is 40%. Ouch. But hold on — the minimum annual cost is significantly less, despite the faster growth rate. In fact you need to spend about $6000 versus the Gold plan above before you start feeling the pain. This means that if you’re relatively healthy and don’t take risks, you’ll likely be ahead of the game with a Bronze plan (which is consistent with conventional wisdom).

In short, when it comes to estimating costs for Silver, Gold, and Platinum plans, a good rule of thumb is to use the sum of the annualized premium plus the deductible as an estimate of maximum annual healthcare costs. Here is how the plans look when sorted by this metric. According to this view, Gold plans are cheapest, followed by Silver. This means that if you are price-sensitive and relatively healthy, the optimal choice is Silver.


This generalization doesn’t hold for Bronze plans because the coinsurance is too high to ignore. An additional caveat is that you have prescription medication or play sports that may take you to the hospital, then you need to look at those costs as a more significant factor.


Plan selection

I only included 39 out of 66 total plans in my analysis. Most of the plans I omitted were plans that included coverage for children or dental. I didn’t include these in order to make the comparison more equitable. I should mention that the MetroPlus plans include Dental and Vision, while the others do not. Here are two contingency tables that show the breakdown across levels as well as provider.

> table(plans$plan.level)

  Bronze     Gold Platinum   Silver
      12       11        3       13

The selection of Platinum plans is not representative primarily because I, personally, am not planning on buying a Platinum plan. If anyone would like to see a more comprehensive analysis, feel free to sponsor more research.

> table(plans$provider, plans$plan.level)

                               Bronze Gold Platinum Silver
  Affinity                          1    1        0      1
  EmblemHealth                      1    1        1      1
  Empire BlueCross/Blue Shield      3    1        1      2
  Fidelis                           1    1        0      2
  Health Republic                   1    2        0      3
  HealthFirst                       1    1        0      1
  MetroPlus                         1    1        0      1
  Oscar                             3    2        1      2
  UnitedHealthcare                  0    1        0      0

Cost of care

The cost of primary and specialist care varies widely. According to one article [3], the average doctor’s visit is $104, while another cites $158. Since I live in New York City, I used $150 for a primary care visit and $325 for specialist. This is codified in a lambda.r type.

Ledger(primary.cost=150, specialist.cost=325) %as% {
  list(cost=0, deductible=0, primary.visits=0, primary.cost=primary.cost,
    specialist.visits=0, specialist.cost=specialist.cost)

Data for each health insurance plan was sourced from NY’s online health exchange. Below is the implementation for determining the cost of each visit based on the previous visits.

add_visit('primary', ledger, plan) %as% {
  if (!$primary.cost) &&
        ledger$primary.visits <= plan$primary.limit) {
    cost <- plan$primary.cost
  } else if (ledger$deductible <= plan$ {
    cost <- ledger$primary.cost
    ledger$deductible <- ledger$deductible + cost
  } else {
    cost <- ledger$primary.cost * plan$coinsurance.pct
  ledger$cost <- c(ledger$cost, cost)
  ledger$primary.visits <- c(ledger$primary.visits, 1)
  ledger$specialist.visits <- c(ledger$specialist.visits, 0)

add_visit('specialist', ledger, plan) %as% {
  if (ledger$deductible <= plan$ {
    cost <- ledger$specialist.cost
    ledger$deductible <- ledger$deductible + cost
  } else {
    if ($coinsurance.pct))
      cost <- ledger$specialist.cost
      cost <- ledger$specialist.cost * plan$coinsurance.pct
  ledger$cost <- c(ledger$cost, cost)
  ledger$primary.visits <- c(ledger$primary.visits, 0)
  ledger$specialist.visits <- c(ledger$specialist.visits, 1)

Calculating the total cost is then simply a fold operation.

Sampling paths to create a surface

As mentioned above, as the number of visits increase, the number of ways to get to that particular combination of visits increases. The number of paths is defined by

p = \left\{ \begin{array}{ll}  m & \mbox{if } n = 0 \\  n & \mbox{if } m = 0 \\  mn + 1 & \mbox{otherwise}  \end{array}  \right.

where m is the number of doctor visits and n is the number of specialist visits. Hence the point with the most paths is when there are 20 doctor visits and 20 specialist visits, with a total of 101 possible paths. Theoretically this shouldn’t be an issue and the surface should be smooth. However, in my implementation (and probably in the real-world), the crossing of the deductible is not necessarily exact. Hence a portion of a payment that theoretically should be discounted may not be discounted. This is why the surface gets lumpy once the cost exceeds the deductible. While I didn’t verify it, presumably the maximum deviation then is going to be based on the highest cost, or c_{\mbox{specialist}} - 1 = 324.


As this post shows, it takes a fair amount of work to estimate healthcare costs. The main takeaway is that if you are healthy, a good way to estimate your annual maximum healthcare bill is to sum the annualized premium with the deductible. The cost savings over a few years will likely pay for any catastrophic healthcare, although this will be less valid the older you get. If you or your family require a lot of care, then you need to estimate the number of visits per year and whether you expect any hospital visits.

In part 3 of this series, I’ll unveil a web application that performs these calculations in the browser, so that it is easier to estimate your healthcare costs. If there is enough interest, I’ll add more factors into the model so that it is more precise.


[1] How do I choose Marketplace insurance? Visited 13 February, 2014.
[2] Out-of-pocket maximum/limit. Visited 13 February, 2014.
[3] Doctor Visit Costs. Visited 14 February, 2014.
[4] Guide to understanding health care costs. Fallon Health. Visited 14 February, 2014.

Why healthcare exchanges are not well-functioning markets


, , , ,

This is part 1/3 in my series on healthcare costs. The analysis is driven by my own needs as an entrepreneur to find affordable healthcare. Part 1 introduces the problem with health exchanges, part 2 addresses the issue of price transparency, and part 3 introduces a web app that estimates your annual healthcare costs.

Politicians like to believe that if you build a marketplace, then a market economy will manifest, delivering all its glorious benefits. What they don’t seem to acknowledge is that for markets to work they need to be well-functioning, which means that the market must satisfy (most) of the conditions of pure competition [1, 2]. If these conditions are not satisfied, then it becomes significantly less likely that the invisible hand of Adam Smith will manifest, nor will prices reach fair value. Some of the (ideal) conditions that must be met are:

  • Infinite buyers and sellers
  • No barriers of entry and exit
  • Perfect information
  • Zero transaction costs
  • Homogeneous products
  • No externalities

In practice, perfect competition cannot be attained given idealized conditions, yet there are reasonable approximations. The foreign exchange markets are generally considered to be the closest approximation of perfect competition. However, in health exchanges none of the above conditions, even in their approximate form, are satisfied. If a market is not well-functioning, then you cannot expect prices to be fair. Hence the claim that bringing health insurance onto a marketplace will increase competition is only half-true. In the sense that there will be more sellers, competition has increased, yet this does not imply that prices will reach equilibrium. Let’s see why.

Infinite buyers and sellers

The first thing to notice is that there are many buyers and few sellers. This is because it is difficult to create an insurance company. Not only do you need a lot of capital, but you have to create a network of providers that will accept your insurance. When there are few sellers in a market, collusion and price fixing can occur. This happens surprisingly often in industries like airlines [3, 4], semiconductors [5], and even perfume [6]. When looking at the premiums of healthcare plans in the same metal level, it is surprising how close the prices are.

No barriers of entry and exit

The difficulty of creating an insurance company is clearly a high barrier to entry for sellers. The consequence is that a seller has the opportunity to unilaterally control prices since competition is limited. In financial terms, this is known as cornering the market. While not necessarily illegal, it can obviously cause price distortions, not to mention lead to antitrust cases. Another consequence is that poor-performing providers can stay in business due to the lack of competition. This means there is a disincentive to improve since revenue can be generated at a lower cost (due to lower spending on providing quality services).

What about buyers? In theory, anyone can enter a market to buy an insurance product with ease. Technology glitches notwithstanding, this is largely a non-issue for buyers, except in the area of transaction costs, which is discussed below.

Perfect information

By perfect information, all participants (both buyers and sellers) are supposed to have full knowledge of the price, utility, quality, etc. of the products [7, 8]. It is very difficult to know the total cost of healthcare based on the premium, deductible, and the out-of-pocket limit alone. So despite there being transparent price in terms of the components of a plan, there is no price transparency when it comes to all the factors driving the total cost of the plan. This is like buying a car based on the price of the tires. In order to know the true cost of your health insurance not only do you have to factor in the premium, the deductible, the copay and/or coinsurance, medication, etc., you also need to estimate the quantity and cost of medical services required per annum. For the average individual this exercise by itself is quite challenging. The only people who really have a good sense of the total cost of healthcare are the health insurance companies themselves. This observation alone shows that information is asymmetric in the market since the sellers have a clear advantage of information over the buyers. The unfair playing field in health insurance is no less egregious than insider trading in finance.

No externalities

An externality is any cost or benefit unintentionally incurred by a third party based on a given activity [9]. For healthcare this takes the form of the cost of service for medical providers. Depending on whether your doctor is in-network or out-of-network in a given plan affects the price that they will charge you. The same is true if you are uninsured [10]. Hence, for the same service and same doctor, the cost of service can vary dramatically. The fact that these prices are opaque means that it is difficult for the cost of medical services to find equilibrium. With health insurance being derived from the cost of services, how can insurance prices possibly be transparent? This is analogous to buying an option on an equity without knowing the value of the underlying stock (startups, anyone?).

In another sense, health insurance can be seen to rely on a different externality: the composition of participants affects everyone’s rates. This is codified in the desire to get as many young, healthy people on the exchange so they can subsidize the cost of the elderly and the sick [11]. Hence, for the same insurance plan, your cost of insurance could go up (down) if other people became sicker (healthier).

Zero transaction costs

What happens when you want to switch plans? Any money spent towards the deductible is lost. Similarly most plans don’t prorate the deductible based on when you sign up. So there is a significant transaction cost to switching, implying that there is a disincentive to switch, which reduces the effects of competition. Compare this to auto insurance or home insurance where it is easy to switch at any time and any premium paid is refunded at a prorated amount.

Homogeneous products

When all products are the same, the product itself becomes fungible. It doesn’t matter which specific product you receive since they are identical. While an individual insurance plan is the same (think same plan, different policy number), they are not fungible across plans since plan details are different. This is similar to how cars are sold in broad categories like luxury, sedan, SUV. To know whether or not you’re getting a good deal requires a lot of research and also knowing the total cost of ownership of the car. One aspect of car buying is that people are generally not rational because there is a large emotional component to automobile purchases. This is not necessarily a bad thing and partially justifies a heterogenous market (since people value the emotion differently). In terms of health insurance, it is unlikely that anyone is going to get excited and have an emotional attachment to a platinum plan.


While it may not be necessary for health exchanges to meet all the conditions of a well-functioning market, it is important to understand the implications of not meeting the conditions. Certain externalities and barriers to entry are probably desirable, but other conditions, such as access to information, need to be improved. Only then can prices begin to reflect the true cost of healthcare and resources allocated efficiently. What is needed is a method to distill the total cost of healthcare down to a single number. This number should represent the best estimate of the cost of healthcare for an individual or family based on each of the plans available. Then it will finally be possible to reliably compare plans.

In part 2, I show how to estimate the annual cost of healthcare based on numerous plans available on the NY Health Exchange. While the examples are specific, the lessons are broadly applicable to anyone looking at health insurance.


[1] Perfect competition. Wikipedia. Visited 14 February, 2014.
[2] Perfect competition. The Economist. Visited 14 February, 2014.
[3] Swiss commission fines airlines for price-fixing. Visited 14 February, 2014.
[4] British Airways fined £58.5m for fuel price-fixing. Visited 14 February, 2014.
[5] Semiconductor Execs Indicted for DRAM Price Fixing. PC World. Visited 14 February, 2014.
[6] Perfume giants fined 40 mn euros for price fixing. Fashion Mag. Visited 14 February, 2014.
[7] Information. Econlib. Visited 14 February, 2014.
[8] Imperfect Information. EconPort. Visited 14 February, 2014.
[9] Externality. Wikipedia. Visited 15 February, 2014.
[10] How to Get Health Care While Uninsured. The Billfold. Visited 14 February, 2014.
[11] Obamacare enrollment push for the young enters 11th hour. Reuters. Visited 15 February, 2014.

A desert motif in ice


, , , , ,

This last shoot in this series is located at 4 Mile Point, part of Scenic Hudson’s conservation project. It’s a modest parcel of land but has nice river access. While taking all these photos of the ice floes, it occurred to me that the barrenness of the landscape is reminiscent of the desert landscape. I thought it would be interesting to explore this idea a bit.

The desert motif

One aspect of the desert motif is the blinding and unrelenting sun overhead. Coupled with an endless barren landscape, the isolation can be awe-inspiring.


Another element of the desert motif that reinforces the feeling of isolation and solitude is the lone tree. Here’s my take.


Perhaps there was life here once, but now…


The cargo ship as freight train

Just as the railroads cut through these barren places, the cargo ships along the Hudson can be imagined serving a similar purpose. To the right is the same ship from earlier. Clearly these ships cannot move particularly fast as they cut through the ice.


While standing on the ice, I took a video of the barge and tug breaking up the ice. I love the sound of the ice cracking and breaking. It’s not like a glacier calving, which sounds like a gunshot. This is more of a long, resonant vibration, like a whip cracking in slow motion. As the ship pushed the ice to the side, you could hear the sound of the crack moving through the ice. It was a bit exhilarating, particularly since I was standing on the ice, and the crack was heading towards me!


Studies of bark, ice, and moss

Wrapping up these pictures, I present a mini study of the trees along the bank. What’s fascinating is that every possible ice formation can be found along the bank, from large sheets, to thin, crystalline patterns, to icicles, to perfectly formed crystal-clear blocks.


I guess as the tide pushes up the ice, some of it gets lodged in strange places, like in exposed tree roots. Here you can also see some dormant moss, waiting for a dash of liquid moisture to rejuvenate.preview.DSCF2495

If I recall, this is a particular species of maple tree that has the multi-colored peeling bark. I love the texture and juxtaposition with the ice.




Get every new post delivered to your Inbox.

Join 205 other followers